Latest / EDGE AI POD / What If A Pair Of Glasses Could Read Intent?

Transcript

What If A Pair Of Glasses Could Read Intent? · EDGE AI POD

• 1:03
  SPEAKER_00: In the world, we have many people affected by neurological conditions that have heavily that are heavily that experience heavily motor dysfunctions.
• 1:18
  SPEAKER_00: So consider, for example, people affected by multiple sclerosis.
• 1:22
  SPEAKER_00: And globally we have more than 2.8 million of individuals suffering of this disease.
• 1:30
  SPEAKER_00: And yet most of them retain the ability to move their eyes.
• 1:36
  SPEAKER_00: So we have this small window of control.
• 1:39
  SPEAKER_00: We can use these eye movements to enable them to restore the communication between their mind and external world.
• 1:49
  SPEAKER_00: And that's why brain computer interface at the beginning started with assistive applications.
• 1:55
  SPEAKER_00: Now it's evolving also with gaming applications and much more.
• 1:59
  SPEAKER_00: But the very first purpose was assisting people.
• 2:03
  SPEAKER_00: And obviously, we have many challenges in assisting people because they struggle with interacting with the external world but also with the technology.
• 2:14
  SPEAKER_00: Because many times the person has to adapt to the system rather than the opposite.
• 2:20
  SPEAKER_00: And moreover, it's very difficult to grab these signals accurately, to process them in a real-time way efficiently.
• 2:30
  SPEAKER_00: So there are very many challenges to be faced.
• 2:35
  SPEAKER_00: We already have commercially many solutions, and we can divide them into main categories.
• 2:41
  SPEAKER_00: On one side we have invasive solutions, and I'm quite sure that many of you already know the Neuralink project by Elon Musk, but there are also BrainGate, Synchrone, and on the other side we have non-invasive techniques such as emotive, neurosky, muse, open BCI.
• 3:02
  SPEAKER_00: On one side, the invasive techniques require a surgical procedure to install these devices into your brain.
• 3:10
  SPEAKER_00: So you can imagine that they are not for everyone, that are designed for specific use cases, and they are very expensive.
• 3:18
  SPEAKER_00: So they cannot be uh they are not affordable for everyone.
• 3:22
  SPEAKER_00: But they are very precise, they have high accuracy, and they can be used for complex tasks such as control a robotic arm in real time.
• 3:34
  SPEAKER_00: And uh but they pose also some ethical uh questions, because Elon Musk, for example, wants to use Neurlink in a first uh phase just for medical application, but then his uh purpose is to uh cognitive augment our brain.
• 3:51
  SPEAKER_00: So let's talk about this.
• 3:53
  SPEAKER_00: Maybe we have some medical concerns.
• 3:56
  SPEAKER_00: And uh on the other side we have non-invasive uh techniques, and uh obviously they are much less accurate than the invasive systems, they are not working in real time, but they are cheap, uh, they have uh they are very easy to uh to use.
• 4:16
  SPEAKER_00: And uh with this work that I'm going to present you, we want somehow to fill this gap because we would like to have a real-time system working on a cheap uh on a cheap device.
• 4:30
  SPEAKER_00: And uh this is our aim.
• 4:32
  SPEAKER_00: And uh for the time being, we are working on a non-invasive system and uh we are focusing on a gaming application, but our aim is to possibly exploit it uh on a system device.
• 4:45
  SPEAKER_00: This is these are the main limitations of uh non-invasive uh BCI systems up to now.
• 4:51
  SPEAKER_00: Uh most of the processing is happening outside the board.
• 4:55
  SPEAKER_00: So you have very limited uh processing happening on board, most of it is happening on a PC or on the cloud.
• 5:03
  SPEAKER_00: And uh moreover, they have very limited real-time performances and they target very narrow applications and that are related to mindfulness, meditations, they analyze the brain waves.
• 5:19
  SPEAKER_00: We started our uh our approach uh with uh uh we started our technique at the beginning using the helmet by op from OpenBCI, and I can assure you that it is quite uncomfortable because it has some pins that uh are in your forehead, so it's not so comfort.
• 5:38
  SPEAKER_00: And uh, but now we are switching uh uh to use uh our biosensor developed by ST together with an STM32 microcontroller where the whole processing happens.
• 5:52
  SPEAKER_00: So the advantage uh of this system with respect to what is available in the market is that it runs completely on an edge device so and it runs in real time, as I will show you later.
• 6:06
  SPEAKER_00: But let's go into the details of the system.
• 6:11
  SPEAKER_00: So our uh our project is composed by a biosensor developed by ST, then a microcontroller STM32H7 that runs the whole uh process, and then the final command is sent to APC for a gaming application that I will show you later on uh on a video.
• 6:33
  SPEAKER_00: But what's the principle of the biosensor?
• 6:37
  SPEAKER_00: It is based on the electrostatic uh charge sensing principle.
• 6:41
  SPEAKER_00: So when we have an object, it is usually uh neutral from the electrostatic standpoint uh because uh its number of uh electrons equals its number of protons.
• 6:54
  SPEAKER_00: But when we they touch and then separate, there are the electrons that uh that move from one uh object to another one.
• 7:02
  SPEAKER_00: And this is uh these are electrostatic charge that create fluctuations in the electric field.
• 7:09
  SPEAKER_00: So this sensor, this biosensor, measures this change, uh this change, and it uses electrodes to feel this change in the electric field and an electronic circuit to amplify and process the signal.
• 7:25
  SPEAKER_00: How we used these uh these electrons, okay.
• 7:28
  SPEAKER_00: This is one of my colleagues that volunteered for this task, and uh we exploited the different configuration for the electrodes, and uh but uh we decided to uh to use uh the eye movements because, as I already said, many people retain the possibility to move the eyes.
• 7:46
  SPEAKER_00: And also we uh decided to use this configuration because we could exploit it in wearable glasses.
• 7:54
  SPEAKER_00: Uh because we exploited also other configurations that could be suitable for visors.
• 7:59
  SPEAKER_00: And uh sometimes they are most effective.
• 8:02
  SPEAKER_00: But uh at uh the at this time we are going uh we are targeting wearable glasses, so we decided to put the electrodes on the nose and uh uh the electrodes in uh behind the here is as a reference.
• 8:15
  SPEAKER_00: We collected the data set using uh with nine people and we decided to grab four different classes.
• 8:22
  SPEAKER_00: The first one is uh unvoluntary blink, but because just because we want to discard it, because we don't want to control anything with our involuntary action.
• 8:32
  SPEAKER_00: Then there is the voluntary blink that can be used like a click of the mouse, and then we have uh uh look looking right and looking left.
• 8:41
  SPEAKER_00: I mean we are looking to the center, then we look right and come back without moving the head.
• 8:47
  SPEAKER_00: This is quite challenging, and then look left and come back.
• 8:52
  SPEAKER_00: And then we designed a completely software pipeline.
• 8:56
  SPEAKER_00: We started obviously designing the whole system on the PC, and then we deployed it on the firmware on our microcontroller.
• 9:04
  SPEAKER_00: And uh this uh pipeline is composed by different blocks that I'm going to explain in details.
• 9:11
  SPEAKER_00: The first block is a pre-processing one.
• 9:13
  SPEAKER_00: Then we have in parallel a data collection block and an event detection, and finally we have an event classification that is based on a one-dimensional convolutional neural network.
• 9:25
  SPEAKER_00: The final class that is recognized is sent via USAT to the PC for controlling the game.
• 9:33
  SPEAKER_00: So, why the pre-processing?
• 9:35
  SPEAKER_00: Because data are quite noisy.
• 9:38
  SPEAKER_00: And uh we are using uh three different uh filters for pre-processing the signal.
• 9:44
  SPEAKER_00: First of all, we are using an high pass filter to remove the DC component, then we are using a notch filter to remove the frequency around 50 Hz to remove the power supply's frequency, and finally we are using a low pass filter with a cutoff frequency of two hertz for removing extra noise that is uh in uh uh in uh harmonic at higher frequencies.
• 10:09
  SPEAKER_00: And here there is a simple of raw signals on the left uh before any processing, any pre-processing, and on the right you can see the effect of applying these filters to the signals.
• 10:22
  SPEAKER_00: So you can see that uh the uh the signal is much is centered around the zero, and uh moreover it's uh much uh less affected by noise.
• 10:32
  SPEAKER_00: And this is helpful uh to have also a small neural network after, because the neural network doesn't have to learn the noise, so can learn directly the uh the clean signal.
• 10:46
  SPEAKER_00: After this uh this pass, we have an event detection.
• 10:50
  SPEAKER_00: We could have uh used directly a temporal uh convolution network for processing directly the signal, but uh it would have been always on.
• 11:01
  SPEAKER_00: We decided to have a very simple algorithm, always on, that is this event detection, and then it activates the neural network for the processing only when there is a relevant signal.
• 11:13
  SPEAKER_00: And this event detector is based on Z score, it is a well-known solution.
• 11:18
  SPEAKER_00: We didn't invent anything here.
• 11:20
  SPEAKER_00: Uh and uh Z score is uh tells you how far is your signal from the average uh in terms of standard deviations.
• 11:30
  SPEAKER_00: And uh consider that in this example uh you uh okay, maybe it is visible, okay.
• 11:36
  SPEAKER_00: Here, uh the event detection doesn't recognize perfectly the peak.
• 11:40
  SPEAKER_00: Uh because where is the point?
• 11:42
  SPEAKER_00: Okay, the peak.
• 11:43
  SPEAKER_00: It recognizes, for example, this part.
• 11:45
  SPEAKER_00: This is because it is working in real time.
• 11:48
  SPEAKER_00: So we don't have the whole the whole signal.
• 11:51
  SPEAKER_00: Uh uh, in this moment we have just uh this part of the signal.
• 11:56
  SPEAKER_00: So the Z score doesn't know that uh the signal is going to grow.
• 12:01
  SPEAKER_00: But this is sufficient to understand that something uh is happening, something different that uh than the normal behavior.
• 12:09
  SPEAKER_00: So when we detect this uh event, we collect uh one uh 100 samples before and 200 samples before for a total of 300 samples.
• 12:22
  SPEAKER_00: And consider that in the firmware implementation, these 100 samples are already grabbed because we have implemented a FIFO mechanism.
• 12:33
  SPEAKER_00: Okay, after that we have a very very simple neural network.
• 12:38
  SPEAKER_00: It is composed by very common layers: convolution, max pooling, uh, real RELU and softmax.
• 12:44
  SPEAKER_00: And as you can see, its uh input is uh 300 because it is the size of our window, by two because we have two channels.
• 12:53
  SPEAKER_00: And the results we gained on this limited uh dataset are quite promising because we obtained about the 90% of accuracy using a K-fold cross-validation procedure, and uh the flash is just uh 18 kilobytes, the RAM six kilobytes.
• 13:12
  SPEAKER_00: And the more interesting part is that this CNN runs in just 0.76 milliseconds.
• 13:21
  SPEAKER_00: But we need much more time to grab the signal because we have 300 samples at 240 hertz.
• 13:28
  SPEAKER_00: That means 1.25 seconds to collect the window and 0.75 milliseconds to execute the network.
• 13:37
  SPEAKER_00: So it is real time.
• 13:40
  SPEAKER_00: And now I can show you the demo.
• 13:43
  SPEAKER_00: Okay, so here we have again uh Vincenzo, my colleague, that is wearing these electrodes, and uh he will uh show he will command this video game.
• 13:54
  SPEAKER_00: So when he looks uh here you can see the commands that is identified.
• 14:00
  SPEAKER_00: When he voluntary blink, the car uh jumps to avoid the obstacle.
• 14:07
  SPEAKER_00: And when when he look uh right the the car moves right, hopefully, if it is correctly understood.
• 14:17
  SPEAKER_00: And the purpose is uh to get the stick coins along the way.
• 14:21
  SPEAKER_00: So let's see.
• 14:23
  SPEAKER_00: Okay, he did uh the score here is incrementing.
• 14:28
  SPEAKER_00: Okay, so this was just a fun uh application, but to understand if the system is working.
• 14:35
  SPEAKER_00: Now our aim is to embed this system in wearable glasses and to make uh real-time testing to receive also feedback.
• 14:45
  SPEAKER_00: Uh, because uh we saw also that uh brain waves and also electrooculography can vary from one person to another.
• 14:53
  SPEAKER_00: So we need to explore also some system of personalization.
• 14:58
  SPEAKER_00: So maybe a calibration procedure will be needed to adjust the weights of the network.
• 15:05
  SPEAKER_00: But uh yes, we are in this process of exploring more, and uh yes, we would like also to expand the functionalities.
• 15:13
  SPEAKER_00: Thank you.