Latest / EDGE AI POD / Smart Sampling Unlocks Edge AI Capabilities You Never Thought Possible

Transcript

Smart Sampling Unlocks Edge AI Capabilities You Never Thought Possible · EDGE AI POD

• 0:09
  Speaker 1: Hi everyone.
• 0:10
  Speaker 1: This is Ankur, co-founder and CEO at Lightscline.
• 0:13
  Speaker 1: With me I also have Ayush Goel sitting here, who is the co-founder and CTO, and we are going to be here for all three days, so looking forward to discussing how we can contribute to the community and what use cases we can work on, and so.
• 0:26
  Speaker 1: So the founding story of LightSkline is actually based on a very counterintuitive insight which is very relevant for the audience of this conference.
• 0:35
  Speaker 1: The insight is basically a simple question does more data actually mean more information?
• 0:41
  Speaker 1: And that's not necessarily the case.
• 0:44
  Speaker 1: In fact, if we look at the human brain, there is something called as the selective attention mechanism, which means that, even in today's presentation, you will just keep the important things in your mind and probably ignore the rest, and so the brain is very efficient at what sort of information it parses.
• 1:01
  Speaker 1: And we essentially thought can we take this question and teach AI models to sort of identify important subsets of data in large streams of sensor data?
• 1:11
  Speaker 1: It turns out that's the case.
• 1:12
  Speaker 1: We are able to get great test accuracies with just 10% of the raw data.
• 1:16
  Speaker 1: So we are going to, today, focus on how we have accelerated some of the sensor AI models using LightSkline AI's SDK, some of the sensor AI models using LightSkline AI's SDK.
• 1:28
  Speaker 1: So the problem we are trying to solve is we are essentially generating huge amounts of sensor data across different modalities, anywhere from space, air, land and sea.
• 1:35
  Speaker 1: So it involves satellite data analysis, undersea fiber optics cables, industrial internet of things and so on.
• 1:42
  Speaker 1: Now this leads to two major problems more infrastructure costs and more human capital costs.
• 1:47
  Speaker 1: Infrastructure costs essentially comprises of cloud computing costs, storage costs, transmission bandwidth requirements and so on, and on the human capital side, it's more.
• 1:58
  Speaker 1: Around every additional hour of data that we are collecting that leads to 40 plus hours of analysis.
• 2:04
  Speaker 1: So you collect five gigabytes of data, and then data teams need to spend a whole lot of time figuring out how to do manual processing and things of that sort.
• 2:14
  Speaker 1: So if we dig a little bit deeper into this, we basically again have different data modalities, different sensor modalities, generating terabytes of data per day.
• 2:22
  Speaker 1: Now what this creates is we start at the edge or at theabytes of data per day?
• 2:25
  Speaker 1: Now what this creates is we start at the edge or at the point of data generation?
• 2:28
  Speaker 1: We are again limited by what size of compute we can put at the edge itself, so we are limited by things like the maximum milliampere hour rating, how much wattage supply can we provide, what is the battery weight, and so on.
• 2:43
  Speaker 1: Next, we have to deal with the bandwidth.
• 2:45
  Speaker 1: So, of course, if we want to, if we need to transmit everything, that has its own cost and complexities.
• 2:51
  Speaker 1: Next is the cloud compute, which is again the cost and complexity of cloud compute is becoming more and more obvious, especially with AI training workloads and with the GPUs and so on.
• 3:01
  Speaker 1: And finally, the human capital component, which, in our experience, has been very similar to the order of magnitude of the infrastructure cost, because each additional hour of data leads to several additional hours of analysis.
• 3:14
  Speaker 1: And finally we get to the decision based on this current pipeline.
• 3:20
  Speaker 1: So how does this pipeline get affected as we get to the terabytes of data per day?
• 3:27
  Speaker 1: How scalable or feasible is this pipeline of essentially flowing the data through this edge compute, bandwidth, cloud compute, storage, human capital cost and so on?
• 3:37
  Speaker 1: And we need to start thinking more about what are the end-to-end requirements of getting to a decision from the raw data.
• 3:43
  Speaker 1: So dealing with more data has actually pushed us to the has just kept us in the more flops or the higher data volumes and in the cloud realm, whereas what we want to do, and especially the theme of this conference, is we want to push more use cases to the edge.
• 4:00
  Speaker 1: So the entire argument is I will show how we have taken three use cases around variables, industrial Internet of Things and distributed fiber optic sensing, which is a new emerging capability, and how we have moved those capabilities both training and inference from Cloud all the way to the Edge.
• 4:20
  Speaker 1: So now, if we imagine a future like this, so this can be on Mars, this is essentially visualizing physical intelligence, lightweight intelligence.
• 4:28
  Speaker 1: There's no way we can just put additional GPUs and we can build bigger data centers and so on.
• 4:34
  Speaker 1: In fact, physical intelligence requires different form factors because we may want to deploy intelligence on different sites.
• 4:40
  Speaker 1: So robots require a different form factor, variables require a different form factor, and so on.
• 4:45
  Speaker 1: So how do we build AI models which are specific to those, the form factor capabilities?
• 4:52
  Speaker 1: And this comes back to the original question of how can we do more with less, and this is where we can take some inspiration from the brain around how can we do novel algorithmic development to exploit structure that is already present in real world sensor data?
• 5:06
  Speaker 1: So in this respect, this is a snippet from IEEE Spectrum, which did a story on one of our research recently.
• 5:14
  Speaker 1: The question we ask here is can we be more selective?
• 5:17
  Speaker 1: Instead of collecting everything, can we start collecting what is relevant, and can we train different neural network, new types of neural network architectures to do that?
• 5:28
  Speaker 1: So this is essentially the key insight that we have, which is does more data actually mean more information?
• 5:36
  Speaker 1: That is not true.
• 5:37
  Speaker 1: So you can look at two examples.
• 5:38
  Speaker 1: One on the time series a thousand data points can still be represented by just one data point if we take an appropriate transform.
• 5:45
  Speaker 1: Similarly, in images, we know that even if we discard 95% of the FFT coefficients, we still can get good inferences because there's not a lot of loss of information.
• 5:56
  Speaker 1: So the foundational insight again is that there's a lot of structure in the data.
• 5:59
  Speaker 1: If we exploit that well, we can unlock 100x efficiencies.
• 6:04
  Speaker 1: In fact, past this year, three out of the four Nobel laureates from like there was a talk in Google DeepMind three out of the four Nobel laureates mentioned something like.
• 6:15
  Speaker 1: The next big wave of research will be on how do we exploit the structure in the data, not focus on the models.
• 6:21
  Speaker 1: But there is a lot that can be done at the data level and that can lead to significant efficiency in locks, and so that is pretty much what LightSkline has been building up to so far.
• 6:33
  Speaker 1: So, to summarize, so far, we are making the argument that if we exploit the structure in the data, we can train models that are more than 100x lighter than current models and which are faster, and we can still maintain the prediction accuracies, the performance and so on.
• 6:50
  Speaker 1: So this is some of the media coverage on the paper that we had in nature, which essentially shows a new neural network architecture wherein we got 435x reduction in the number of flops in comparison to convolution neural networks on a variety of data sets.
• 7:07
  Speaker 1: So we'll also look at the universality of this approach on different modalities like variables, industrial distributed fiber optic sensing and so on, like wearables, industrial distributed fiber optic sensing and so on.
• 7:20
  Speaker 1: So what LightSclient does?
• 7:23
  Speaker 1: I would like to introduce this concept of useful flops.
• 7:25
  Speaker 1: So everyone is talking a lot about flops.
• 7:26
  Speaker 1: We see AI labs reporting, we train models on teraflops and gigaflops and all that.
• 7:30
  Speaker 1: Why are we not thinking about what are the useful flops?
• 7:33
  Speaker 1: It's almost like if I give an oil and gas analogy.
• 7:36
  Speaker 1: It's like saying I have a million gallons of crude oil, but what I actually care about is extracting one gallon of gasoline like high power gasoline.
• 7:45
  Speaker 1: So we should have more measures of intelligence that actually focus on how much intelligence or how many use cases we are solving, not on like how many flops can I make my model or how big can I make a model.
• 7:57
  Speaker 1: So that's the foundational argument and we'll see how this 400x reduction has actually pushed use cases like variables, industrial Internet of Things and fiber optics from the cloud to the edge all the way training and inference everything on the edge.
• 8:13
  Speaker 1: To summarize the value prop we are again leveraging AI to automatically train models and run inference on 90 percent less data, which results in lighter and faster models.
• 8:22
  Speaker 1: There are two advantages to this.
• 8:24
  Speaker 1: One is a cost-based, one is a value-based advantage.
• 8:27
  Speaker 1: So cost-based is essentially we reduce 90 percent of the AI in front, people-related time and cost.
• 8:33
  Speaker 1: Value-based is we enable applications that are currently unobtainable, and we'll walk through a lot.
• 8:36
  Speaker 1: We enable applications that are currently unobtainable and we'll walk through a lot of these applications, especially on the distributed fiber optics variables where we have deployed on less than 264 kilobytes of RAM, and so on.
• 8:46
  Speaker 1: So now we'll look at three use cases of how LightSkline AISDK has accelerated both model development and deployment.
• 8:58
  Speaker 1: Throughout the presentation, I'd like to emphasize that this is valid both during training and inference, and that is why this is different than your conventional down sampling or dimensionality reduction, because it translates to the inference, the real time inference stage as well.
• 9:12
  Speaker 1: On the variables, we look at how we have deployed sensor AI inference on less than 264 KB of RAM, so on a Raspberry Pi Pico, which is basically a bare metal device.
• 9:23
  Speaker 1: We'll also look at a Fortune 150 case study wherein we did cloud training with basically 90% less data.
• 9:32
  Speaker 1: We look at a use case on one of the largest software providers on how we achieved 381x less flops and 85x faster training on their datasets.
• 9:43
  Speaker 1: And then, finally, distributed fiber optic sensing is very interesting.
• 9:46
  Speaker 1: So recently there were incidents of fiber cable damage in the Baltic Sea, and so this is gaining a lot of traction wherein, like passive sensing within, distributed fiber optics can be used to extract a lot of information.
• 9:57
  Speaker 1: In passive sensing within distributed fiber optics can be used to extract a lot of information in environments for use cases like traffic monitoring, urban infrastructure monitoring, critical infrastructure monitoring, earthquake prediction.
• 10:07
  Speaker 1: This is also being used by oil and gas companies to do exploration of where to dig for oil wells and so on.
• 10:14
  Speaker 1: So this is the first use case.
• 10:16
  Speaker 1: Here we are looking at a Fortune 150 case study who used our SDK inside their own platform and they are essentially looking at high-capex industrial machine monitoring.
• 10:27
  Speaker 1: So, as you can see, these are like high-capex assets, so bearings, gearboxes, centrifugal pumps and so on.
• 10:35
  Speaker 1: These have a variety of sampling frequencies.
• 10:36
  Speaker 1: So, again, very high sampling frequency, large amounts of data collected within a very small amount of time.
• 10:41
  Speaker 1: Now you can see the accuracies here.
• 10:43
  Speaker 1: They achieve pretty remarkable accuracies, but they train using just 10 percent of the raw data.
• 10:48
  Speaker 1: So think about if you had one GB of data per use case.
• 10:52
  Speaker 1: We are actually sampling and working or training our models only on the 10% of that data.
• 10:57
  Speaker 1: We are never training on everything.
• 10:59
  Speaker 1: Now how do we do the sampling and stuff?
• 11:01
  Speaker 1: That is all the IP and that is how the smart sampling happens, because we cannot naively down sample, because then we use the Nyquist information.
• 11:08
  Speaker 1: But this is to show a customer success.
• 11:11
  Speaker 1: Study on like this essentially reduces a whole lot of their cloud computing costs because we are again training only on the 10 percent of the relevant data.
• 11:20
  Speaker 1: Another validation they showed on how this is better than any conventional downsampling is.
• 11:26
  Speaker 1: They compared our approach with principal component analysis, which is one of the very common dimensionality reduction techniques.
• 11:32
  Speaker 1: Here you can see original and light-scline curve overlap on each other, whereas PCA is way off.
• 11:38
  Speaker 1: So to summarize what this means is that our sampling and model training combined together preserves the information, whereas if we just naively downsample or use other dimensionality reduction techniques it does not.
• 11:51
  Speaker 1: And the other thing is the other way this is different than any of the dimensionality reduction approaches is we sample during training.
• 11:59
  Speaker 1: But this also means that while inference we are only running inference on a few samples.
• 12:04
  Speaker 1: So that is how we achieve significant benefits.
• 12:06
  Speaker 1: And the 400x flops reduction also happens during the inference stage, which is a big deal.
• 12:13
  Speaker 1: Next case study is with one of the largest privately held software providers.
• 12:17
  Speaker 1: So here you can see that we again took one of their use cases.
• 12:21
  Speaker 1: We compared here with traditional ML, with feature engineering, traditional ML with raw data, so compared a lot of existing approaches, also deep neural networks like CNNs, which are at least still now state-of-the-art in these kinds of data.
• 12:37
  Speaker 1: And then we used LightScline, so both sampling and training combined.
• 12:41
  Speaker 1: But here with that we again achieved best state-of-the-art accuracies with 381x less flops, 85x faster than current models.
• 12:51
  Speaker 1: So for them, the value prop again is less cloud cost, faster training times, data scientists not having to spend a lot of time on feature extraction and looking at all of the data streams, and so on.
• 13:00
  Speaker 1: A lot of those things can be automated.
• 13:02
  Speaker 1: Now this slide talks about the universality of the approach.
• 13:08
  Speaker 1: So these results are again from the Nature paper, wherein we showed results on different modalities like variables, industrials, dealing with wide variety of sampling frequencies and so on.
• 13:18
  Speaker 1: Because one of the questions can be how universally applicable is this and do we have to do a lot of fine-tuning or things like those types of things, depending on the data?
• 13:28
  Speaker 1: The answer is no.
• 13:29
  Speaker 1: You can still use the same four lines of code to work on a variety of use cases, which is what we have been able to do.
• 13:35
  Speaker 1: So these are four different datasets which are like industry standard datasets.
• 13:40
  Speaker 1: We validated them for both industrials and variables use cases and achieve different amounts of benefits in terms of how many flops we need, while maintaining the prediction accuracies.
• 13:54
  Speaker 1: Let's see if I can play this video.
• 13:55
  Speaker 1: Yeah, I think.
• 13:59
  Speaker 1: How should I be able to play this video?
• 14:07
  Speaker 1: There's a video here.
• 14:08
  Speaker 1: Yeah, there it is.
• 14:11
  Speaker 1: So this is the SDK itself.
• 14:13
  Speaker 1: So it starts with.
• 14:15
  Speaker 1: It's basically a wheel file, so you download that into your laptop.
• 14:19
  Speaker 1: We next this section is importing the data.
• 14:21
  Speaker 1: So here we look at 51 channels of variables time series data and then we'll show the dimensionality here.
• 14:30
  Speaker 1: So once the data is ingested, then we will call LightSkline's sampling and training module together.
• 14:37
  Speaker 1: So this is the data loading process.
• 14:41
  Speaker 1: The data itself is big.
• 14:42
  Speaker 1: This data is approximately 1.5 gigabytes in size.
• 14:46
  Speaker 1: So this next block is going to ingest that data in the LightScline compute module and then this reduce data and train model is the key function which, if you can see that this does basically the sampling and the model training together, then you can see the loss going down.
• 15:03
  Speaker 1: So standard, like PyTorch framework here this is again compatible with your standard machine learning libraries in terms of PyTorch, keras SQL.
• 15:14
  Speaker 1: So not a lot of work on your end.
• 15:16
  Speaker 1: It's very easy to use that.
• 15:17
  Speaker 1: Then, finally, I think the video got finished but we also showed the training accuracies that we were able to get 99% training accuracies.
• 15:28
  Speaker 1: So the main benefit here was this training actually happens in literally 75 seconds.
• 15:36
  Speaker 1: If we were to use all the raw data, if we were to do manual feature extraction.
• 15:40
  Speaker 1: That would be like several hours, like 20, 30, 40 hours, to do all of those things.
• 15:45
  Speaker 1: But now, because we are training neural networks that can actually figure out what data to collect based on a loss function, we can actually significantly accelerate both the training and deployment phases.
• 15:58
  Speaker 1: Next we will look at the distributed fiber optic sensing modality.
• 16:03
  Speaker 1: So this data is actually very large.
• 16:05
  Speaker 1: These are almost like images, but each data is essentially a time series, and then you have several channels, because fiber optic cables can be several hundred kilometers in length.
• 16:16
  Speaker 1: So we are working on some datasets which are more than 50 kilometers of fiber optics data.
• 16:21
  Speaker 1: These can typically generate a few hundred gigabytes of data, depending on what length of cable.
• 16:28
  Speaker 1: We are monitoring and dealing with this volume of data on a daily basis.
• 16:32
  Speaker 1: In fact, these are on very remote environments, so think about, like offshore environments or traffic monitoring scenarios.
• 16:40
  Speaker 1: So we don't even have the capability to transmit all of that data.
• 16:44
  Speaker 1: We don't have a very powerful server located on site to process that.
• 16:48
  Speaker 1: So that itself, right now, the biggest, literally the biggest challenges for customers, even to send us data, is how do we send this to you?
• 16:55
  Speaker 1: Do we send this to you in a hard disk, do we send it to you in a nas server and so on, because this is, this is something like that cannot be even sent on a OneDrive link.
• 17:04
  Speaker 1: So one of the customers literally had 8 terabytes of data just from a few days of fiber optic sensing and so just sending that data.
• 17:12
  Speaker 1: What sort of infra do we need for that?
• 17:14
  Speaker 1: That is a challenge, but this is an emerging application.
• 17:16
  Speaker 1: It's growing very rapidly.
• 17:17
  Speaker 1: So here we were able to train again by sampling both in time and across channels.
• 17:23
  Speaker 1: We were able to train these models in under eight minutes on an Intel i7 processor.
• 17:28
  Speaker 1: But again, by exploiting the structure in the data, we don't necessarily have to build very complex models.
• 17:34
  Speaker 1: We can again reduce the size of the models, make the training and inferences much faster.
• 17:42
  Speaker 1: Now we'll start looking at some use cases of training on the edge itself.
• 17:46
  Speaker 1: So here we are looking at how the training went on two edge devices.
• 17:51
  Speaker 1: So we looked at NVIDIA Jetson Nano and Intel i7.
• 17:55
  Speaker 1: So on the left you can see that if we use Jetson Nano, if we use more than 50% of the raw data, we cannot do training on Jetson Nano because it runs out of resources.
• 18:05
  Speaker 1: However, we can do training on Jetson Nano if we actually use just 10-20% of the raw data, because that turns out to be enough RAM and compute.
• 18:14
  Speaker 1: Intel i7 is more powerful, so that's why we can still do on that.
• 18:18
  Speaker 1: If we look at the transfer learning scenario, so in that case again, transfer learning is only possible on the edge if we use light-screen compute, meaning that again, with 100% of the raw data, jetson Nano is not able to do transfer learning.
• 18:32
  Speaker 1: So I had some questions on can we do on-device learning or self-learning?
• 18:36
  Speaker 1: This is an example of that.
• 18:38
  Speaker 1: So think about if person A has a specific signature of walking, we want to adapt that to a person B, and so we can do that capability on the device itself.
• 18:47
  Speaker 1: So we don't have to do the retraining run on the cloud.
• 18:49
  Speaker 1: We cannot retrain the last two layers, essentially on the device itself, and so that is how we are going to enable on-device learning.
• 19:00
  Speaker 1: This is an example of the smallest form factor we have deployed our AI in.
• 19:04
  Speaker 1: So this is Raspberry Pi Pico, which is 264 KB of RAM, a few milliwatts of power, and so we have been able to deploy.
• 19:12
  Speaker 1: This is actually running inference on that.
• 19:13
  Speaker 1: So we are collecting time series data and then running inference on top of that, on top of that, so these would be the benefits, which is this can enable training, inference and transfer learning on your current chips, because it unlocks a lot of power.
• 19:31
  Speaker 1: And finally, also newer products.
• 19:33
  Speaker 1: They can have inbuilt self-learning capabilities.
• 19:36
  Speaker 1: And to end with, I think we partitioned the audience in two ways.
• 19:41
  Speaker 1: One is chip companies or integrated device manufacturers, and one is the system integrators, so happy to work with you.
• 19:47
  Speaker 1: You can scan the QR and put in if any use case comes to your mind.
• 19:51
  Speaker 1: Me and Ayush both will be here, so we'll be happy to discuss over the next few days, thank you.
• 19:56
  Speaker 1: Thank you, ankur.
• 20:03
  Speaker 1: So we have time for some quick questions from Ankur.