Latest / EDGE AI POD / Applying GenAI to Mice Monitoring

Transcript

Applying GenAI to Mice Monitoring · EDGE AI POD

• 0:06
  Speaker 1: There are three aspects in this talk that I would like to suggest you to consider.
• 0:12
  Speaker 1: First of all, for what edge-generative AI can be useful.
• 0:17
  Speaker 1: I will focalize to only one use case.
• 0:21
  Speaker 1: Now, these use cases why is representative Is representative because behind that there is a problem which is affecting the operations of some small or even medium medium enterprises.
• 0:41
  Speaker 1: And then the next topic is yeah, but how to generate data in a way that the problem can be shaped and captured properly.
• 0:57
  Speaker 1: And the third is yeah, but once the algorithm, the pipeline, is shaped, on which edge processor to deploy it and what type of features these edge processors should feature.
• 1:12
  Speaker 1: So, essentially, generative AI is today dominated by large enterprises, and fixed function AI is what we are working on in this community since 2018, 2019.
• 1:42
  Speaker 1: So imagine being a small-medium enterprise which took years to get familiarized with fixed-function AI detectors and the like, and now generative AI comes into the picture.
• 1:57
  Speaker 1: That can easily be confusing Not large enterprises, small and medium enterprises.
• 2:08
  Speaker 1: So that's all about.
• 2:10
  Speaker 1: I cannot turn.
• 2:14
  Speaker 1: The I said is not working.
• 2:15
  Speaker 1: Okay, yeah, it's going to be a long talk, okay, so, yeah, next, just the next slide, please.
• 3:17
  Speaker 1: So the business is not large, I mean, and is quite flat.
• 3:23
  Speaker 1: It will slowly grow over the next five years.
• 3:28
  Speaker 1: So, from a money point of view, you may not find very interesting, but still these are small, medium enterprises.
• 3:38
  Speaker 1: So try, do we want to scale generative AI?
• 3:43
  Speaker 1: We need to care about this type of company.
• 3:45
  Speaker 1: Then you may like or may not like the problem, but that's the point I mean.
• 3:52
  Speaker 1: The point is how to get them into the picture of edge generative AI.
• 3:57
  Speaker 1: Okay, so next, and these guys, one of these companies, already developed fixed-function AI.
• 4:09
  Speaker 1: On which type of target?
• 4:10
  Speaker 1: Well, these are cages.
• 4:12
  Speaker 1: The cages are hosting a small amount of mice and they are used for research, experimentation, and the trend is to sensorize the cage with imagers, with microcontrollers.
• 4:33
  Speaker 1: To achieve what fixed function AI?
• 4:42
  Speaker 1: Which means that we are focused on low power processing, on analyzing the data locally and provide some assessment.
• 4:50
  Speaker 1: What type of assessment Next?
• 4:52
  Speaker 1: So that's the dashboard.
• 4:57
  Speaker 1: Essentially, these are different cameras which are capturing.
• 5:02
  Speaker 1: You may run, please, no, sorry, the previous one, you may.
• 5:07
  Speaker 1: Essentially, in the dashboard, you have different views of the cage and here the exercise is to understand what is the amount of water available, what is amount of the food available and if the water is available in the cage, because companies are very concerned about their wellness, so the mice wellness is truly at the top of their priority.
• 5:40
  Speaker 1: So fixed AI essentially is used there Classifier, even detector, the next one.
• 5:46
  Speaker 1: And also lighting conditions may be very poor, because imagine these cages are in a rack.
• 5:56
  Speaker 1: The rack typically is 90, 100 cages, and then you have several cages Placed where, placed in clean rooms, because any type of contaminants should not affect the life of this guy, and the value of these mice increases as soon as the big pharma proceeds on the experimentation.
• 6:22
  Speaker 1: So that's fixed AI and fixed function AI, so you may use your kind of YOLO or convolutional neural network to classify and so on.
• 6:31
  Speaker 1: Next, but then why do we need generative AI?
• 6:35
  Speaker 1: Because we need to essentially move at the next level, process the information such that we can create new information, assess the past information and create new information.
• 6:50
  Speaker 1: What type of information?
• 6:52
  Speaker 1: Information that allows a natural machine interaction kind of application.
• 6:58
  Speaker 1: So you may smile because what does it mean?
• 7:01
  Speaker 1: I want to interact in a natural way with mice.
• 7:04
  Speaker 1: Well, with their behavior.
• 7:06
  Speaker 1: Yes, that's the evolution, because the fixed function AI is already in place.
• 7:11
  Speaker 1: So we need to move forward, and always having in mind their wellness.
• 7:22
  Speaker 1: Here energy-efficient processing is the key.
• 7:25
  Speaker 1: So what is the problem?
• 7:28
  Speaker 1: Essentially, the problem is you can display the whole.
• 7:31
  Speaker 1: Yeah, essentially the problem is you can display the whole.
• 7:33
  Speaker 1: Yeah, essentially the problem is I want to assess the past information, visual information in very short amount of time for example, 20 frames and then predict the next intent of the mice, such that I can predict their behavior.
• 7:48
  Speaker 1: If I can predict their behavior especially these guys may have a weird behavior, especially when they are many in the same cages I can protect them.
• 8:01
  Speaker 1: And you know, predicting this information is important because you cannot enter in the clean room at any point in time.
• 8:07
  Speaker 1: You have to wear, you have to be very decontaminated.
• 8:10
  Speaker 1: So this type of processing becomes important.
• 8:16
  Speaker 1: Resolution is VGA is enough, frames per second are four and we had to work on data set creation.
• 8:26
  Speaker 1: For example, in this case we captured 3.6 million of frames and these are example of the frames, like in the detector, like in the detection, and really the mice can be placed anywhere.
• 8:42
  Speaker 1: In this case we just placed with one cage we want to simplify the problem and with one mice.
• 8:51
  Speaker 1: Now this is the pipeline.
• 8:52
  Speaker 1: Let me explain.
• 8:54
  Speaker 1: This is the pipeline to achieve labeling, because all these images need to be labeled.
• 9:00
  Speaker 1: So imagine that I have on top a textual query.
• 9:06
  Speaker 1: Then this pipeline, which essentially is composed by two parts, the bottom part is creating a representation which is handcrafted.
• 9:19
  Speaker 1: In which way?
• 9:20
  Speaker 1: Essentially, we use standard deviation to understand the type of motion which is into the past frames 16 frames and we detect the level of motion, we detect also if these are sleeping, and then we create a first one-hot encoding.
• 9:41
  Speaker 1: Then, okay, you can move forward.
• 9:46
  Speaker 1: The next, the next I think I say this the next, then the upper part of the pipeline.
• 9:53
  Speaker 1: Instead, use a state-of-the-art algorithm such as VideoLava, which generates draft labels and then these are combined with the encoding coming from the uncrafted pipeline and then these are further processed by Lama3.
• 10:10
  Speaker 1: And at the end, I have labeled all the images.
• 10:14
  Speaker 1: Therefore, this has been created and now I have these 3.6 millions of frames which have been labeled.
• 10:26
  Speaker 1: Next one so what type of pipeline are we devising?
• 10:31
  Speaker 1: Essentially, the pipeline is the first one.
• 10:35
  Speaker 1: The baseline is called video-to-action.
• 10:36
  Speaker 1: Video-to-action is composed by three pipelines Video-my, which is state-of-the-art 114 million parameters.
• 10:45
  Speaker 1: And adapter, which is a fully connected layer 0.6 million parameters.
• 10:50
  Speaker 1: And then OPT, open, pre-training, transform 125 millions of parameters.
• 10:58
  Speaker 1: Then on these we played essentially frames coming, the 16 past frames and then the intents are predicted out of this pipeline.
• 11:09
  Speaker 1: Now, one exercise was to decrease the complexity of the pipeline.
• 11:14
  Speaker 1: By the way, you see the order of magnitude it's essentially 230, 240 million parameters.
• 11:21
  Speaker 1: We don't need gigaparameters or even more.
• 11:27
  Speaker 1: That's an example of the scale we are looking for for edge generative AI.
• 11:32
  Speaker 1: And then we simply stripped some part of the Video my in order to save 62% of the parameter for the Video my and achieved something like 170 million parameter.
• 11:47
  Speaker 1: So there were two versions.
• 11:49
  Speaker 1: The next and then training was composed by three steps.
• 11:54
  Speaker 1: Essentially it was an incremental training.
• 11:57
  Speaker 1: In the first step we wanted to train Video my in the fully connected layer, while OPT was frozen.
• 12:05
  Speaker 1: In the second one you go next.
• 12:09
  Speaker 1: In the second one, we used parameter-efficient fine-tuning in order to fine-tune OPT.
• 12:18
  Speaker 1: And in the third step there was a bundle adjustment of all the weights for the three networks in order to achieve the final operation.
• 12:27
  Speaker 1: That was not much time-consuming because, for example, we used the Hugin-Face capability to run this training.
• 12:39
  Speaker 1: So I didn't need a supercluster AI or a super-powerful supercomputer to run these things because, once again, from the perspective of a small-medium enterprises, this guy cannot afford a Corvex or Colosseum supercomputer.
• 12:56
  Speaker 1: Simply, there is no budget for that.
• 12:59
  Speaker 1: Okay, so the next one.
• 13:01
  Speaker 1: So these are the performances.
• 13:04
  Speaker 1: We went down 29% in terms of trainables.
• 13:08
  Speaker 1: In terms of trainables, accuracy by the tiny V2A was improved by 1.86%.
• 13:15
  Speaker 1: Validation and testing, and the perplexity was decreased by 37%.
• 13:25
  Speaker 1: So these are the key parameter to validate and assess the quality of the neural network.
• 13:33
  Speaker 1: The transformer, the next one.
• 13:34
  Speaker 1: And then what type of target did we use?
• 13:37
  Speaker 1: Well, I'm from ST, so it was not an ST processor, not for the moment.
• 13:42
  Speaker 1: It was a Raspberry P, p4, P5, obviously.
• 13:48
  Speaker 1: So the type of processor 2.4 gigahertz quad processor with a lot of LLPR DRAM.
• 13:55
  Speaker 1: That's not a microcontroller, it's a multiprocessor.
• 13:59
  Speaker 1: So that's the type of architecture we are looking at.
• 14:05
  Speaker 1: Think about it.
• 14:07
  Speaker 1: I did also a survey about implementations in the last years, and the multi-processing unit dominates.
• 14:14
  Speaker 1: That's not the space for microcontroller.
• 14:17
  Speaker 1: There were very few examples.
• 14:19
  Speaker 1: So that's another result of the transformation, and the best results that we obtained were lower than 300 milliseconds per inference, which is not bad.
• 14:32
  Speaker 1: Hardware acceleration at the moment, at my poor knowledge, doesn't exist.
• 14:37
  Speaker 1: If, when and if they will exist, certainly we will go down at least one order of magnitude.
• 14:44
  Speaker 1: You know what does it mean?
• 14:46
  Speaker 1: 30 milliseconds with me, which means real time.
• 14:50
  Speaker 1: Okay.
• 14:51
  Speaker 1: So we are looking for to see maybe someone in this community to develop such devices.
• 14:57
  Speaker 1: The memory is huge 130, but because we use the Onyx runtime.
• 15:02
  Speaker 1: Now, if you strip down the Onyx runtime certainly you can lower in a massive way the amount of rhyme and lowering the amount of RIME and lowering the amount of RIME plus hardware acceleration certainly will make the implementation more energy efficient.
• 15:20
  Speaker 1: So the conclusions Next.
• 15:24
  Speaker 1: So that's the pipeline, okay, you understood.
• 15:29
  Speaker 1: And how to make natural the interaction?
• 15:34
  Speaker 1: At least one way using text-to-speech.
• 15:37
  Speaker 1: What kind of text-to-speech?
• 15:38
  Speaker 1: There is a nice project on GitHub which is Piper.
• 15:42
  Speaker 1: Essentially, the penalty between the length of the audio and the processing time is 40%, which is nice.
• 15:50
  Speaker 1: It's a 15 million parameter generative neural network.
• 15:56
  Speaker 1: So at the end you see that three over four neural networks are generative in this pipeline.
• 16:03
  Speaker 1: The fully connected is just an adapter to adapt the representation between the two neural networks.
• 16:09
  Speaker 1: So we got less than 300 milliseconds on Raspberry Pi 5.
• 16:13
  Speaker 1: Perplexity 2.64.
• 16:15
  Speaker 1: So I think it can fly and in the future will fly even better.
• 16:21
  Speaker 1: What are the limitations of this work?
• 16:23
  Speaker 1: One mice, one cage.
• 16:25
  Speaker 1: Obviously, in the cage usually three to five mice live together.
• 16:31
  Speaker 1: They have more complex behavior.
• 16:33
  Speaker 1: They have their own complex language, so maybe this application is not huge in terms of businesses, but technology-wise.
• 16:44
  Speaker 1: I hope it will give you a sense of the complexity that is behind and the opportunity, which certainly represents one sample of the possible deployment of edge-generated AI, especially from the perspective of small-medium enterprises, which are of paramount importance to achieve and to get.
• 17:08
  Speaker 1: And then I'm done and sorry for these issues.
• 17:13
  Speaker 1: I hope it was smooth.