Latest / EDGE AI POD / An Embedded Transformer- base face recognition system in the STM32N6

Transcript

An Embedded Transformer- base face recognition system in the STM32N6 · EDGE AI POD

• 0:08
  SPEAKER_00: Good afternoon everyone.
• 0:13
  SPEAKER_00: I would like to thank the AGI Foundation for being here and to give me the opportunity to present this work on embedded transformer-based face recognition system on the STM32 and 6.
• 0:23
  SPEAKER_00: So, first of all, I am David Aiello, software engineering, graduated in 2024 at Polytechnico di Torino.
• 0:30
  SPEAKER_00: I'm currently working on real-time AI development on embedded system focusing on harder acceleration computer vision at Sensor Reply.
• 0:39
  SPEAKER_00: So these are the topics that I will cover today, a brief introduction about Reply Sensor Reply, the STM32 and 6 platform, the hardware user for this project.
• 0:49
  SPEAKER_00: We'll take a look at context and challenge, the system path and model architecture performances, and finally the conclusion.
• 0:57
  SPEAKER_00: So Reply is a digital consulting company with a global present, especially in Italy, Germany, and USA.
• 1:03
  SPEAKER_00: On the bottom part, we can see how Reply has grown since 2007 in terms of revenue and people.
• 1:10
  SPEAKER_00: Here we have our competence area that covers every digital field, starting from digital humans, robotic, digital marketing, cybersecurity, and NAI, our partners ecosystems, some of our customers.
• 1:26
  SPEAKER_00: But as I said, I am from Sensor Reply, which is the reply group company specialized in the embedded AI solution.
• 1:34
  SPEAKER_00: Our technology background is built on two main pillars: physical modeling and embedded AI and real-time computing.
• 1:41
  SPEAKER_00: So starting from that, we are able to provide several services like condition monitoring, virtual sensor, real-time image recognition, digital things, and control systems.
• 1:52
  SPEAKER_00: So over the years we have built a portfolio of edge solutions spun across multiple domains, face recognition, gesture recognition, image captioning, and so on.
• 2:01
  SPEAKER_00: We are here today to present a solution that combined the new hardware with the most promising model of the past year, which is a transformer, a model that typically runs on edge platform.
• 2:11
  SPEAKER_00: And thanks to our pattern, ST microelectronics, we were able to get early access to STM32N6, which is the first microcontroller of the ST family, to feature a neural accelerator.
• 2:23
  SPEAKER_00: So this is the STM32 and 6.
• 2:25
  SPEAKER_00: I suppose that you have already seen this board these days.
• 2:28
  SPEAKER_00: It features a neural processing unit, ARM Cortex M55, 4 MB of continuous RAM, is able to provide up to 600 Gig operation, achieving three tops per watt.
• 2:40
  SPEAKER_00: And the development kit also comes with a five inches LCD screen and with a camera camera module.
• 2:47
  SPEAKER_00: And this is the NPU, an accelerator specifically 304A application which is able to offer better efficiency compared to GPU, is easier to use than FPGA and more flexible than ASIC.
• 2:59
  SPEAKER_00: Basically, it's capable of transforming an edge device from a simple sensor into an autonomous and intelligent brain.
• 3:06
  SPEAKER_00: So the context here explore the capability and constraint of the N6 platform, improve and expand the histome face recognition system is something that we already have, and evaluate the implementation of a transforming-based model on a microcontroller.
• 3:20
  SPEAKER_00: So starting from the context, several challenges arise, like deploying multiple models in sequence, our core face recognition model, which is built on a transforming-based architecture, even a spoofing detection using only the device built-in camera, and an internal database that allows users to dynamically enroll into the system.
• 3:40
  SPEAKER_00: So this is the pipeline.
• 3:42
  SPEAKER_00: So we uh capture the frame from the camera, then we have facial detection steps that aims to locate faces into the capture the frame, then the face alignment phase, which is responsible alignment phase of canonical view to improve robustness, the anti-spoofing detection ensure resilience against sprint or video attack, and at the end the core model, the phase recognition, which extract an embedding from a detected face.
• 4:07
  SPEAKER_00: Starting from this embedding, at the end we can uh compare this embedding with the one uh already uh extracted, so the one uh present in the database.
• 4:17
  SPEAKER_00: Those are some of the model architecture for the phase detection which is a retina face that feature uh feature pyramid network, and this means that the network is able to uh detect uh face of different size, is also a multitask model because it's also able to detect the landmarks on the on the face.
• 4:34
  SPEAKER_00: Uh for the anti-spoofing which is mobile net V2, it has been training to uh distinguish between real image and uh spoof image.
• 4:43
  SPEAKER_00: So before uh discussing about uh the core model, the one based on the transformers, I would like to introduce the main difference between uh CNN and transformers.
• 4:52
  SPEAKER_00: So, for example, CNN has typical inarchical structures based on the receptive field, while transforming instead has a global structure based on the attention mechanism.
• 5:02
  SPEAKER_00: Um CNN are uh has effective local feature extraction while transformers are able to model the long distance relationship.
• 5:11
  SPEAKER_00: CNN has a strong inductive bias to convolution operator, while on the other side there is no strength inductive bias from from since thanks to this reason, transformers are able to uh truly understand the pattern inside the data.
• 5:23
  SPEAKER_00: Uh CNN can be reduced with a compromise accuracy.
• 5:26
  SPEAKER_00: For this reason, they're generally computing less uh demanding, while instead transformers perform well typically on an IM number of parameters.
• 5:34
  SPEAKER_00: And so for this reason are computationally intensive.
• 5:38
  SPEAKER_00: So by studying uh by studying the state of the art, we figure out a model trend in lightweight networks.
• 5:43
  SPEAKER_00: So on this table, uh, I put the main network from different families.
• 5:47
  SPEAKER_00: We have a pre-CNN that indicates classical CNN appeared before uh vision transformer, post CNN that denotes CNN that integrates the narrative of VITs but retain a poor CNN structure, and then VITS and uh hybrid networks that are networks that combine both convolution operation and uh attention mechanism.
• 6:07
  SPEAKER_00: And we figure out that only this type of network, this hybrid structure, are able to maintain a low number of parameters while also preserving uh an eye accuracy.
• 6:15
  SPEAKER_00: Uh we can say edge next is the best trade-off between the number of parameters and accuracy.
• 6:20
  SPEAKER_00: So we choose for the last uh model edge face, which is the winner of the efficient face recognition competition in 2023, which is based on EdgeNext, and uh as we can see it has an anarchical structure divided in stages like a classical convolutional neural network, a convolutional encoder, and STDA encoder which is based on the attention mechanism.
• 6:41
  SPEAKER_00: Uh the uh as output it provides us an embedding of 512 value.
• 6:47
  SPEAKER_00: So in order to deploy a Loviz model uh on uh the hardware, we need to rely on the STM32 toolchain, especially STM32 QBI, which has uh recently changed uh its name in STGI, which is a part of the STM32 toolchain, uh, is able to convert as a tool uh TFLite on an X Keras model into a C code for edge execution.
• 7:08
  SPEAKER_00: In this case it was mandatory given the presence of the NPU, but unfortunately not all the deep layers are supported, especially for uh the last uh model.
• 7:18
  SPEAKER_00: Uh in this case we have problems for all of these uh operators and layers, and layer normalization gel activation has been replaced respectively with batch normalization and hard switch, while instead batch matmul, which is the dot product, the core of the attention mechanism, results to be more complex to replace.
• 7:38
  SPEAKER_00: So this is the self-attention mechanism, the classical one.
• 7:42
  SPEAKER_00: We have the fully connected layer at the at the beginning, then we obtain the matrix's uh query, key and values, we multiplied with the inner product the matrix Q and K, and then with the softmax we are able to obtain an attention map, then is already multiplied with the with the matrix value.
• 8:01
  SPEAKER_00: So here the goal is to find a solution which is able to emulate this type of behavior without relying on the inner product.
• 8:09
  SPEAKER_00: And actually, we were able uh we are able to find that the on the convolutional self-attention from the NVIDIA blog.
• 8:16
  SPEAKER_00: Uh as I said, this is designed to emulate the relational encoding process of self-attention, and as we can see, the inner product has been relayed has been uh replaced with the Admart product, which is basically an element-wise multiplication.
• 8:30
  SPEAKER_00: Thanks to the convolution, it enables the modeling of both local and global features uh relying exclusively on convolution.
• 8:37
  SPEAKER_00: It turns out also, according to the experiment conducted by the um the man from NVIDIA to be a faster and also a more accurate than pure transformers architecture.
• 8:49
  SPEAKER_00: Uh we will see uh now in details uh what happens.
• 8:52
  SPEAKER_00: So the structure starts with an initial modeling of local relationship within the image.
• 8:57
  SPEAKER_00: So thanks to this uh that wise convolution, we obtain the matrix uh V.
• 9:02
  SPEAKER_00: Uh then we obtain the matrix Q with a mapping of the channel dimension from C to H times W.
• 9:08
  SPEAKER_00: And then by performing the computation of the matrix uh transf transpose and followed by the reshaping of the matrix Q, we obtain the matrix key.
• 9:17
  SPEAKER_00: So now the important things to notice is that uh uh the matrix uh query for each uh dimension has all of the value of all of the pixels.
• 9:26
  SPEAKER_00: While instead the matrix uh key for a dimension has the value of just one pixel.
• 9:31
  SPEAKER_00: And since they have the same dimension, we can actually multiply that um with the element-wise multiplication, and this enables each pixel to interact with all of uh the other pixel within a specific channel.
• 9:44
  SPEAKER_00: Obtain a sort of uh pixel wise aggregation like in the classical attention mechanism.
• 9:50
  SPEAKER_00: Then we are able also to make this aggregation strong uh stronger by projecting it into a new space thanks to the pointwise convolution, and the sigmoid emulates the softmax, uh, allowing us to obtain a weighted uh weighted map.
• 10:05
  SPEAKER_00: And then at the end, this weighted map is element-wise multiplied with the matrix value, reproducing the second phase of the attention.
• 10:14
  SPEAKER_00: So uh these are the performance we are able to obtain for the full pipeline uh around uh 40 milliseconds and execution speed which is around 55, sorry, 25 fps.
• 10:27
  SPEAKER_00: So those are the inference times comparison, the the parameters for each model, uh the inference times for the STM32N6, the STM32H7, and also for the desktop.
• 10:38
  SPEAKER_00: Um as we can see the presence of the NPU play a crucial role.
• 10:44
  SPEAKER_00: We are able to obtain a speed up of 78 uh times for mobile face net compared to STM STM32H7, which is the most powerful microcontroller available on um on the cloud.
• 10:56
  SPEAKER_00: Uh we also compare the mobile face net because it's the classical uh network used for face recognition on Edge.
• 11:05
  SPEAKER_00: And this is the comparison in terms of accuracy.
• 11:08
  SPEAKER_00: As we can see, Edge Face, our model, proves to be more effective uh than mobile face net on all validation data sets.
• 11:16
  SPEAKER_00: So this type of solution can be embedded in a wide range of scenarios where user identification is required, such as access control, uh, to restrict our sensitive areas, granting me permission or privilege, seamless identification, and also personalized user experience in private or control environments.
• 11:33
  SPEAKER_00: So the future perspective in this case is reuse of the transformant-based architecture for other computer vision tasks on microcontroller.