Latest / EDGE AI POD / Neural-ART: ST’s New NPU Architecture at the Edge

Transcript

Neural-ART: ST’s New NPU Architecture at the Edge · EDGE AI POD

• 0:06
  SPEAKER_00: Thanks a lot for the honor to be able to present here uh Neural Art, so ST's new MPU architecture.
• 0:14
  SPEAKER_00: And uh well, maybe some of you had the chance to see downstairs the STM32N6 demos.
• 0:21
  SPEAKER_00: Um obviously, there is this type of MPU integrated.
• 0:26
  SPEAKER_00: Maybe some of you are also curious what is under the hood actually.
• 0:30
  SPEAKER_00: I will provide here some technical information and give you an idea what we are trying to do in the future.
• 0:39
  SPEAKER_00: So, NeuralArt was not born just in one day.
• 0:44
  SPEAKER_00: It was a long journey, so it took us almost 10 years to go through various prototypes, trials and errors, and many measurements to actually being able in 2024 to bring it to the public and release uh this STM32N6 together, also with some we saw a rather wave of new uh devices coming out with MPUs integrated at that time.
• 1:13
  SPEAKER_00: So now I would like to show you what we are actually doing, and during that time we learned certain things.
• 1:21
  SPEAKER_00: One of them is we learned that our MPU had to be somehow flexible.
• 1:27
  SPEAKER_00: So it was not enough to have a single network that we could execute, it needed to somehow be programmable.
• 1:34
  SPEAKER_00: Another important topic was scalability.
• 1:38
  SPEAKER_00: So we knew we had different products, different uh let's say, scales of performance.
• 1:43
  SPEAKER_00: So we need to have an architecture which is somehow scalable.
• 1:47
  SPEAKER_00: Another very important issue was the fact that on edge devices memory bandwidth is a scarce resource.
• 1:54
  SPEAKER_00: So no DDR, and you have to be very careful with it.
• 1:58
  SPEAKER_00: And last and not least, silicon area is an important topic because it defines how many dyes you can sell per waiver, and therefore the pressure was obviously there.
• 2:09
  SPEAKER_00: So, how does that look like?
• 2:12
  SPEAKER_00: This is the STM2N6, and it has an ARM, um M55 core integrated, neural art accelerator, quite some decent SRAM on chip.
• 2:25
  SPEAKER_00: Um it has GPU, it has uh ASP and a lot of interfaces, obviously.
• 2:31
  SPEAKER_00: Now, what is this MPU actually doing?
• 2:33
  SPEAKER_00: It's a stream-based architecture.
• 2:36
  SPEAKER_00: That means we have a centralized stream switch in the middle that is routing data among different components.
• 2:45
  SPEAKER_00: And this all at the same time.
• 2:48
  SPEAKER_00: That means there is no interaction between the streams or blocking.
• 2:52
  SPEAKER_00: There are DMAs, a set of DMAs, their job is to translate data tensors that are somewhere stored in memory into data streams and route them through the stream to the accelerators that are integrated.
• 3:04
  SPEAKER_00: These accelerators have different functionalities like convolution, pooling, activation, and many other things.
• 3:10
  SPEAKER_00: Support, also different precision, 8-bit, 8-bit, mixed position, 16, 16-bit, and so on.
• 3:17
  SPEAKER_00: And we have additional U components like for debugging, we have uh on-the-fly compression, encryption engines, interrupt generation, temporary storage, plus also this thing on the lower right, which is a programmable controller that allows us to execute all these programming of the system independently from the host.
• 3:39
  SPEAKER_00: So we are offloading everything to the MPU.
• 3:43
  SPEAKER_00: Now, how do you program that?
• 3:47
  SPEAKER_00: On the left side, you see a classical representation, an Onex graph of a network.
• 3:54
  SPEAKER_00: We have a specific compiler that translates that into a graph where the graph is segmented in epochs.
• 4:01
  SPEAKER_00: Epochs means these are the units that are connected at the same time and being alive at the same time.
• 4:09
  SPEAKER_00: These epochs are scheduled one after the other on the um MPU itself.
• 4:16
  SPEAKER_00: And this is handled either by host or by the epoch controller.
• 4:19
  SPEAKER_00: Now these epochs can be quite complex, they can be alive for several million cycles depending on the complexity of the network.
• 4:28
  SPEAKER_00: And there can be also multiple of these processing chains at the same time, or they can be depending on the amount of resources that you have on your MPU configuration, they can be more complex or less complex.
• 4:41
  SPEAKER_00: So, now as I said, memory is a scarce resource, so we have to be careful.
• 4:47
  SPEAKER_00: And the advantage of such a stream-based architecture with uh data streams and chains of processing units is that we don't have to go always forth and back to memory.
• 4:59
  SPEAKER_00: So we route calculations to the different functional units.
• 5:03
  SPEAKER_00: Now you can do that in different ways.
• 5:05
  SPEAKER_00: Example A, for example, there we are using two parallel chains.
• 5:10
  SPEAKER_00: Example B, we use one chain, just slightly differently uh combined.
• 5:17
  SPEAKER_00: If we do rather complex network layers, we have to somehow iterate across a parameter, is for example, input channels or output channels.
• 5:28
  SPEAKER_00: Depending on how we do that, the effect, however, on memory can be quite significant.
• 5:33
  SPEAKER_00: So be the difference between A and B is that one of them creates three times more memory traffic.
• 5:39
  SPEAKER_00: But be careful, this is just true for this example.
• 5:43
  SPEAKER_00: There are others' examples where it is exactly the opposite.
• 5:46
  SPEAKER_00: So it's the job of the compiler to find the exact best configuration to map the chains in a way that you are using your memory as efficient as possible.
• 6:00
  SPEAKER_00: So then we also learned you said it has to be scalable.
• 6:04
  SPEAKER_00: We have different products, so we actually are not building a single MPU, we are building an MPU generator.
• 6:11
  SPEAKER_00: So it's a tool suite.
• 6:13
  SPEAKER_00: This gets a description of a few tens of lines, a library of components, we run it through it, it produces a hardware description for the compiler, it produces an RTL database, it produces documentation, it produces libraries, software libraries, and so on.
• 6:29
  SPEAKER_00: And then you have different MPUs there.
• 6:31
  SPEAKER_00: So from low end to middle level in the tera op range to the big guys, which are multi-island.
• 6:38
  SPEAKER_00: We call that multi-island if you have multiple switches there.
• 6:41
  SPEAKER_00: And obviously, this has a constraint in terms of size.
• 6:45
  SPEAKER_00: The more resources you integrate, the bigger this thing gets, and the cost is obviously the factor.
• 6:53
  SPEAKER_00: So this is the classical MPU design.
• 6:56
  SPEAKER_00: And there are two things that we also saw.
• 6:59
  SPEAKER_00: We saw that the important figures of merit, so power efficiency and density, they are somehow limited.
• 7:09
  SPEAKER_00: In the power efficiency, there it was somewhere between one and five teraops per watt for the technologies that we can afford for uh an MCU, right?
• 7:19
  SPEAKER_00: So it's obviously not three nanometers in Fed.
• 7:23
  SPEAKER_00: It's somewhere 16 FF or 18 FTSOI.
• 7:28
  SPEAKER_00: On the density, story similar.
• 7:31
  SPEAKER_00: It's somewhere between 0.1 and 2 teraops per square millimeter.
• 7:35
  SPEAKER_00: Obviously, here it depends on what do you count as an operation, right?
• 7:40
  SPEAKER_00: And the learning is that, well, what we think will be needed in the future is a lot more than that, specifically on the power efficiency.
• 7:50
  SPEAKER_00: So what can we do?
• 7:53
  SPEAKER_00: And we got to the conclusion we have to find something new.
• 7:56
  SPEAKER_00: So we are looking at in-memory computing.
• 8:00
  SPEAKER_00: And there are two ways, principally.
• 8:02
  SPEAKER_00: You could do that obviously on in Astroram or in non-volatile memory, but basically you can separate them in digital in-memory computing and analog in-memory computing.
• 8:13
  SPEAKER_00: Digital is quite straightforward, right?
• 8:15
  SPEAKER_00: This is deterministic, it is easy to be integrated in a digital environment, all fine.
• 8:22
  SPEAKER_00: You can also use it as an Astro RAM itself if you are not using it for computing.
• 8:27
  SPEAKER_00: Analog IMC is a bit more tricky because this is approximate computing, it's more efficient for certain parameters, and if you integrate that, you have to be a bit careful because they can have you know some sensitivity on process changes, temperature, and such things.
• 8:45
  SPEAKER_00: So, if we look at now at real data, how does it look like?
• 8:50
  SPEAKER_00: This is data that we generated from a prototype on 18 FTSI.
• 8:54
  SPEAKER_00: So on the upper left, you see a graph of the energy efficiency of such an implementation.
• 9:02
  SPEAKER_00: And there, if you this is obviously dependent on um sparsity and transition, but you see this the black curve is one tile, so one memory that does a computation.
• 9:14
  SPEAKER_00: And the green one is a cluster of eight of these tiles with some logic around it that you need, right?
• 9:21
  SPEAKER_00: And you see you are in the range of 40 terops per watt.
• 9:26
  SPEAKER_00: So this is a 10x for sure better than what we had before.
• 9:30
  SPEAKER_00: This here, this is the density.
• 9:34
  SPEAKER_00: If we look something at one gigahertz here, we are in the range of 10 tops per square millimeter, right?
• 9:43
  SPEAKER_00: Again, more or less 10x better.
• 9:46
  SPEAKER_00: If we compare that with analog, um, the situation is the following that analog is obviously also supporting different widths.
• 9:54
  SPEAKER_00: So there the tendency is to go a bit to narrower number representation, and there, depending on that, we see another factor that could be gained.
• 10:05
  SPEAKER_00: But as I said, analog has some additional catches, right?
• 10:09
  SPEAKER_00: But this is more or less what we expect from this approach.
• 10:13
  SPEAKER_00: So 10 or even more X improvement.
• 10:18
  SPEAKER_00: So then there's a second problem.
• 10:20
  SPEAKER_00: Energy efficiency is not only about how efficient are your tiles, also how you integrate it.
• 10:27
  SPEAKER_00: So this is an example where we integrate such tile for a very simple network, a regular network computation.
• 10:36
  SPEAKER_00: Just 50% of the power dissipation is gone by reloading and data movement between uh these units.
• 10:46
  SPEAKER_00: The other topic is number representation.
• 10:50
  SPEAKER_00: Obviously, the smaller number representation you are choosing, the more power efficient you can be.
• 10:56
  SPEAKER_00: But this is what you spend.
• 10:59
  SPEAKER_00: And are there ways around it?
• 11:03
  SPEAKER_00: Well, we could talk about weight stationary, right?
• 11:06
  SPEAKER_00: So not moving weights around, you keep that local.
• 11:10
  SPEAKER_00: And ST is working since many years on phase change memory, PCM, embedded.
• 11:16
  SPEAKER_00: These are very dense memories.
• 11:18
  SPEAKER_00: So we are talking here about densities that go up to 50 million bits per square millimeter on an 80 nanometer technology, which is very significant.
• 11:29
  SPEAKER_00: And we started to integrate here analog in-memory computing with also cells that can store multiple levels, so even multi-level cells storing multiple bits per cell.
• 11:44
  SPEAKER_00: This is interesting, but it has a catch.
• 11:48
  SPEAKER_00: PCM memories, they are very slow in loading and not very power efficient if you write to them.
• 11:56
  SPEAKER_00: This means it has to be absolutely weight stationary, no really reloading allowed.
• 12:00
  SPEAKER_00: That means for a device, either you have enough memory on it or no mapping for what you want to do.
• 12:08
  SPEAKER_00: Second problem is these memory, they have drifts.
• 12:13
  SPEAKER_00: You have to compensate that.
• 12:15
  SPEAKER_00: This is drift compensation, can be done, uh, but it's a bit more tricky, so you have to be careful with your application.
• 12:23
  SPEAKER_00: So this is another option.
• 12:25
  SPEAKER_00: Again, with advantages and disadvantages.
• 12:29
  SPEAKER_00: We know that none of these technologies is actually perfect, right?
• 12:34
  SPEAKER_00: There's no one-fits-all technology.
• 12:37
  SPEAKER_00: Therefore, we think that it's maybe the right approach to go for something that's heterogeneous.
• 12:42
  SPEAKER_00: So it uses the advantages of the different technologies and exploits that in a unified architecture.
• 12:51
  SPEAKER_00: So there's the problem of number representation, precision, there's the problem of accuracy, there's the problem of throughput, power efficiency, and so on.
• 13:00
  SPEAKER_00: What we try here is we combine the different units in a 2D mesh, and we try to map the parts of the algorithm that are can better deal with one or the other property on the different nodes.
• 13:12
  SPEAKER_00: This is a project, a European financed research project called NeoSock, where we exactly try to do that.
• 13:19
  SPEAKER_00: And the idea is that in next year we will try to tape out a chip on 80 nanometer that should demonstrate this.
• 13:30
  SPEAKER_00: And then I get to my conclusion.
• 13:32
  SPEAKER_00: So on the NeuroArt, it is a flexible data stream processing architecture.
• 13:39
  SPEAKER_00: It's not a CPU, it's a data stream data path.
• 13:44
  SPEAKER_00: Our focus was on cost, memory bandwidth efficiency, scalability, and reconfigurability.
• 13:50
  SPEAKER_00: But we somehow understood that the maximum power efficiency and density is there is somewhere a limit that you can reach with this approach.
• 14:02
  SPEAKER_00: Standard cell-based approach.
• 14:05
  SPEAKER_00: So we are looking at in-memory computing, analog or digital.
• 14:09
  SPEAKER_00: We know that the weight reloading is a problem, it's costly, so weight stationary could be a solution to fix that.
• 14:18
  SPEAKER_00: And as I showed, PCM is an interesting uh option there.
• 14:23
  SPEAKER_00: But maybe in the end, as there will be not a technology that fits all, you will have something that uses a bit of everything.
• 14:33
  SPEAKER_00: So a heterogeneous architecture.
• 14:36
  SPEAKER_00: Thanks a lot for your attention.
• 14:38
  SPEAKER_00: So if there are any questions.