Latest / EDGE AI POD / Powering Intelligence: Anaflash's Revolutionary AI Microcontroller with Embedded Flash Memory

Transcript

Powering Intelligence: Anaflash's Revolutionary AI Microcontroller with Embedded Flash Memory · EDGE AI POD

• 0:07
  Speaker 1: Our team at Anaflash has developed an AI microcontroller with embedded flash memory.
• 0:14
  Speaker 1: We will discuss why we designed it and how it benefits the Edge AI application.
• 0:22
  Speaker 1: The Edge AI application.
• 0:27
  Speaker 1: This is the outline we're going to present today.
• 0:29
  Speaker 1: I will start with a brief background before explaining our design.
• 0:37
  Speaker 1: In particular, I will cover the Neo-Memory Computing Unit, which I will refer to as NMCU, and the waterline driver, which I will refer to as WL driver for embedded flash.
• 0:52
  Speaker 1: Finally, I will present the silicon test result and conclude the presentation.
• 1:03
  Speaker 1: To power the intelligent edge AI is indispensable.
• 1:09
  Speaker 1: However, two key challenges stand in the way power, efficiency and cost.
• 1:20
  Speaker 1: For example, customers want AI models with more computations and higher accuracy, with minimizing the need for frequent battery changes.
• 1:29
  Speaker 1: Every intelligent edge device requires non-volatile memory to store AI parameters and a high-performance yet efficient AI processor.
• 1:43
  Speaker 1: Our technology delivers both, enabling smarter, more sustainable edge computing.
• 1:51
  Speaker 1: Traditional non-volatile memory, such as STT-MNM, demands extra processing steps, increasing cost, while offering only 1 bit per cell capacity.
• 2:10
  Speaker 1: Our solution overcomes these limitations with a seamless, comfortable embedded flash memory that requires no additional masks and scales.
• 2:22
  Speaker 1: Efficiently requires no additional masks and scales.
• 2:25
  Speaker 1: Efficiently delivering a high, efficient and high cost.
• 2:29
  Speaker 1: Efficient and high performance alternative.
• 2:35
  Speaker 1: Let's go on to the architecture of microcontroller.
• 2:38
  Speaker 1: In most edge AI applications, memory bottlenecks are more common than computing bottlenecks advanced packaging technology, including GPUs and memory chips in a single chip to enhance their high-speed interface.
• 3:10
  Speaker 1: To tackle the challenges, we present an AI microcontroller with zero-standby, power-weight memory featuring 4-bit per-cell embedded flash technology which is seamlessly integrated with an accumulated calculators.
• 3:31
  Speaker 1: This is why we call it near memory computing unit, a key.
• 3:40
  Speaker 1: The key benefit of this architecture is tightly coupled with a wide I-O interface between the NMCU and flash memory on a single chip, enabling efficient AI processing.
• 3:57
  Speaker 1: Since FLASH is a non-volatile memory, there is no need to fetch data from external memory after booting or waking up from deep sleep.
• 4:10
  Speaker 1: This feature makes it a strong candidate for many edge AI solutions.
• 4:18
  Speaker 1: We implemented all of this using a standard logic process without any additional steps.
• 4:30
  Speaker 1: Next, let's take a closer look at how the NMCU is operating.
• 4:36
  Speaker 1: The NMCU, based on embedded flash memory, consists of three main components.
• 4:44
  Speaker 1: Embedded flash memory consists of three main components.
• 4:50
  Speaker 1: First, the control logic manages weight addresses in the embedded flash memory and regulates the input-output buffer flow, allowing Mac operations to be executed in a single instruction without CPU intervention during processing.
• 5:04
  Speaker 1: Second, 16 processing elements enhance parallel computations by sharing weights transmitted from each bank of the eight embedded flash macros through a high bandwidth I-O interface.
• 5:21
  Speaker 1: Finally, the last stage block quantizes the 32-bit Mac operations result into 8-bit activation values to reduce the memory size.
• 5:34
  Speaker 1: The control logic is responsible for controlling the flow of ML operations.
• 5:42
  Speaker 1: It automatically manages the addresses of the embedded flash where the weights parameters are stored, as well as the initiation of read operations, thereby minimizing CPU intervention during ML operations.
• 5:59
  Speaker 1: By minimizing CPU intervention during ML operations.
• 6:02
  Speaker 1: Additionally, it allows selecting input from the input buffer and ping pong buffer and includes logic for storing the final activation values from the quantization logic into the ping pong buffer.
• 6:26
  Speaker 1: Ping pong buffer here is the PE clusters that consist of 16 vector processors called process elements.
• 6:29
  Speaker 1: Each PE is implemented with digital logic based on an adder tree and can perform 8 by 1 vector operations in a single cycle.
• 6:40
  Speaker 1: To support partial sum operations, each PE includes a 32-bit accumulation buffer.
• 6:53
  Speaker 1: The quantization logic is a hardware logic designed to quantize 32-bit computation results of the PE cluster into 8-bit format.
• 7:03
  Speaker 1: It supports the relative activation function, along with 32-bit scale and bias, and it can perform shift operations to extract computation results in 8-bit format by combining the 32-bit format.
• 7:20
  Speaker 1: By combining the 32-bit scale and shift operations, fixed-point arithmetic can be implemented.
• 7:28
  Speaker 1: This is how the operations of each block so far are represented in the time domain.
• 7:43
  Speaker 1: First, after transmitting the input data to the buffer, the computation begins.
• 7:49
  Speaker 1: At this point the weight addresses are sent to each bank of the embedded flash where the weight data is stored.
• 7:59
  Speaker 1: After 30 cycles the data reaches the processing element.
• 8:03
  Speaker 1: Since the MAC operations start only when the required weight data, the processing element does not begin computations during the initial 30 cycles needed for weight fetching.
• 8:18
  Speaker 1: However, given that the program times of embedded flash memory is significantly longer than its read time, making it read-only during inference, we designed the system so that, if the same address has been fetched, the MAC operations can start immediately.
• 8:40
  Speaker 1: Can start immediately.
• 8:42
  Speaker 1: The logic for the MAC unit has been optimized to complete computations within 28 cycles, excluding the one cycle needed for quantization and one another cycle needed for write-back from the 30 cycles for weight fetching.
• 9:06
  Speaker 1: In this study, the logic was designed to perform 128 meg operations over 16 cycles.
• 9:13
  Speaker 1: To minimize the memory movement of the input data, we introduced a ping pong buffer of the input data.
• 9:28
  Speaker 1: We introduced the ping pong buffer To start DNN model layer 1 calculation.
• 9:30
  Speaker 1: The input buffer is used for MAC operation and the output value is written back to a ping pong buffer A.
• 9:36
  Speaker 1: The next step when performing MAC operations using the ping-pong buffer, the NMCU automatically considers the data from the buffer where the previous result was written back as new input data and sends it to the processing element.
• 9:58
  Speaker 1: For layers with input sizes that do not exceed the ping-p pong buffer size, computation can begin immediately without additional data transfers, enabling efficient execution For next-layer calculation.
• 10:16
  Speaker 1: Ping pong buffer connections are changed.
• 10:20
  Speaker 1: This makes it possible to continue MEC operations without stop.
• 10:32
  Speaker 1: I will talk about overstress-free double-rail driver circuit published in JSTC 2023.
• 10:39
  Speaker 1: Developed for embedded flash.
• 10:51
  Speaker 1: Has M-MOS only pass to charge from VRE to WL.
• 10:54
  Speaker 1: Because of the VTH drop and body effect of the M-MOS only path, the verified read-rebel for state S15 among 16 QLC embedded flesh cell status has to be much less than VDDH.
• 11:16
  Speaker 1: This results in limited embedded flesh cell margin.
• 11:20
  Speaker 1: To address this issue, we propose an overstress-free WL driver specifically developed for QH embedded flash.
• 11:29
  Speaker 1: Differently from the conventional WL drivers circuit, we have separate boosted supply lines such as VPS1234 and VPS123 lines.
• 11:46
  Speaker 1: Also, pmos charging circuit from VLD has been added to extend S15 verified read reference level up to VDDH.
• 11:56
  Speaker 1: Different level of VPS and VPP are generated by standard logic comfortable high voltage generator separately.
• 12:07
  Speaker 1: Now we are going to check the experimental result.
• 12:15
  Speaker 1: Here is the die photograph of our design, measuring 4 by 4.5 square millimeters and fabricated using Samsung Foundry's 28 nanometer standard logic process.
• 12:30
  Speaker 1: It features two blocks of 4-bit per cell embedded flash to enhance the wide IO interface.
• 12:38
  Speaker 1: Additionally, a tightly coupled NMCU is positioned next to the flash to accelerate parallel computations, while RISC-V processor handles additional operations, along with SRAM and other components.
• 12:55
  Speaker 1: Compared to other works, this design features a unique 4B-per-cell non-volatile memory without process overhead.
• 13:05
  Speaker 1: This slide shows the measurement result of overstress-free WL driver circuit for QLE embedded flash.
• 13:17
  Speaker 1: Measured waveforms show that the selected PWL and WWL lines switch well to the different VLD levels from 0.5 to 2.5 volt successfully.
• 13:34
  Speaker 1: This is the retention measurement result of the fabricated microcontroller chip running AI inference task.
• 13:50
  Speaker 1: Both MNIST and Deep Auto Encoder were tested.
• 13:53
  Speaker 1: You can find that the weight distribution according to the threshold voltage For MNIST case the inference accuracy shows over 95%.
• 14:09
  Speaker 1: It achieves nearly same as software baseline.
• 14:13
  Speaker 1: For deep autoencoder case, it achieves the same accuracy of software baseline as 0.878 AUC.
• 14:21
  Speaker 1: We did a benchmark test by DeepOtto encoder model from TinyMF-Hoff.
• 14:29
  Speaker 1: Our cycle accurate simulate result shows better performance compared to other design.
• 14:35
  Speaker 1: Let's go on to the summary.
• 14:40
  Speaker 1: Embedded non-volatile memory is essential for smart edge device to reduce power consumption.
• 14:48
  Speaker 1: Conventional embedded non-volatile memory is expensive, while density is not good, so we propose an AI microcontroller with embedded flash which is fabricated in 28 standard logic.
• 15:04
  Speaker 1: Thank you for all support for this work.