Latest / EDGE AI POD / Edge AI From Atoms To Apps with Dr. Mohamed Sabry of Nanoveu

Transcript

Edge AI From Atoms To Apps with Dr. Mohamed Sabry of Nanoveu · EDGE AI POD

• 0:07
  SPEAKER_00: Okay, um Thanks everyone for sticking around.
• 0:10
  SPEAKER_00: Um my name is Mohammed Sabrul Mohammed Ali, and I'm very excited to be here today or I share with you what we're doing.
• 0:17
  SPEAKER_00: Um I'm losing a little bit of my both hats.
• 0:20
  SPEAKER_00: So I'm a professor and I'm founder of this company.
• 0:23
  SPEAKER_00: So I will try to leverage both talents to try to give some nice speech.
• 0:27
  SPEAKER_00: Um I will try to grab also a lot of things in 15 minutes, so uh bear with me if there is any kind of difficulty, but I'll try to make it as enjoyable as possible.
• 0:37
  SPEAKER_00: So the idea of the title was I'm going from apps to from atoms to apps, right, and try to capture that we look into things from the nanotechnology level all the way to the application and systems level when we design what we have in EMAS.
• 0:51
  SPEAKER_00: So, what is EMAS is actually a fabulous semiconductor company that spun off out of the research, uh, my research group.
• 0:59
  SPEAKER_00: We're trying to build very efficient, right, edge AI system on chip devices that looks into all the benefits of nanotechnology devices, overcoming their limitations at the system level with scalable solutions.
• 1:12
  SPEAKER_00: So the idea here is that very high energy efficiency and scalability at the same time.
• 1:18
  SPEAKER_00: We can the chip that we have now called the XDOT, can give you around 20 times improved energy efficiency, and this has a lot of use cases from wearables to industrial IoT, as I will explain later.
• 1:31
  SPEAKER_00: We now we're founded in Singapore, but now we've actually expanded across the globe.
• 1:36
  SPEAKER_00: So we have in Australia, Singapore, Egypt, and the United States, right?
• 1:41
  SPEAKER_00: All of this are our operational offices, RD offices, and so on.
• 1:45
  SPEAKER_00: And we have all the expertise from application development all the way to chip level design and even nanotechnology device modeling and optimizations.
• 1:55
  SPEAKER_00: All right, so the leadership team, mean while I'm being the founder, I'm now the CEO of the company, Mark Gorenson, who is um came out of retirement to join the company, become the CEO of the NA of the semiconductor branch of it.
• 2:09
  SPEAKER_00: And Scott Spiser is the vice president for sales and marketing.
• 2:12
  SPEAKER_00: And today, actually, we have a booth.
• 2:14
  SPEAKER_00: Um honored to have also mean uh Dr.
• 2:16
  SPEAKER_00: Sergio and Nikila, who are also one of the lead engineers in my team.
• 2:22
  SPEAKER_00: All right, so let's take a look here at uh what we have, right?
• 2:26
  SPEAKER_00: So the journey is always from when you design a chip, right, we start from the application system level requirements, and then we'll look into what kind of devices we can use to deliver, right, what we need.
• 2:39
  SPEAKER_00: Okay, and the key point is it has to be mature technology, so the founders have to give it to you, right?
• 2:44
  SPEAKER_00: It doesn't have to be it has to be scalable, but it doesn't have to be fully yielded.
• 2:49
  SPEAKER_00: We can overcome some of the limitations in it.
• 2:52
  SPEAKER_00: So when we take this, then you can develop some sort of modules, right?
• 2:56
  SPEAKER_00: Because the chip itself doesn't work alone, so you need to come up with some sort of board design and also have some sort of innovative aspects to it, right, to make sure that it can have many use cases.
• 3:07
  SPEAKER_00: Then we see how that fits into the application budget and then how we can develop your software to leverage all the benefits that you have at the chip level.
• 3:15
  SPEAKER_00: So going all full circle, right, from the algorithms to the architecture to the circuit design to the technology, we'll always find these hidden knobs in between them.
• 3:25
  SPEAKER_00: And when you tweak them, you achieve huge benefits that's not possible by taking each of these layers of abstraction by itself.
• 3:33
  SPEAKER_00: So the chip that we have that we have developed is called the XDOT, is a full system, right, with a RISC-V as a processor core that orchestrates all the data flow movement and can run an operating system on top of it.
• 3:44
  SPEAKER_00: Now we have instead of one accelerator, we have two AI accelerators.
• 3:48
  SPEAKER_00: Each one is tweaked and optimized for specific kinds of vector matrix operations, right?
• 3:54
  SPEAKER_00: So depending on what kind of application you have, it will select one of each of these two automatically for you through the compiler.
• 4:02
  SPEAKER_00: Then we also have a lot amount, I mean, a decent amount of memory from non-volatile and even SRAM for your temporary variables.
• 4:10
  SPEAKER_00: But you cannot have enough memory, right?
• 4:13
  SPEAKER_00: Always applications can um overflow that.
• 4:16
  SPEAKER_00: So what we do is that we develop a certain optimization technique that compresses your AI workload, and we have an AI weight decompressor that allows you to decompress the weights, right, before even execution.
• 4:29
  SPEAKER_00: So you achieve the combined benefits of reduced memory footprint and high accuracy in your AI classification.
• 4:35
  SPEAKER_00: You don't have to sacrifice that.
• 4:37
  SPEAKER_00: Then when you have all of this together and you can run it, you can actually achieve always on inferencing with around two milliwatts as an average power, can go up to five milliwatts if you account for IOs and whatnot.
• 4:49
  SPEAKER_00: And the chip itself actually could be packaged in a 5 plus 5 QFN, but if you're doing in a chip level package CSPs, it can go to around uh four multiplied by three, so way, way smaller than we have right now.
• 5:03
  SPEAKER_00: So this was the chip, but it's also complemented by a lot of like research, especially in the software layers, right?
• 5:10
  SPEAKER_00: So, how we can make this this edge chip run actually meaningful AI applications when these AI applications usually are quite huge.
• 5:18
  SPEAKER_00: So the key point that you want to make these things smaller, and we look at all the possible aspects of it, from tuning to quantization, um knowledge distillation, pruning, right, and even uh coding as well.
• 5:32
  SPEAKER_00: All of this combined together can get the models to even a hundred times smaller than its original memory footprint.
• 5:38
  SPEAKER_00: And suddenly, instead of using hundreds of megabytes, you need now a few hundreds of kilobytes to run this kind of model.
• 5:44
  SPEAKER_00: So this allows us to run many of the let's say conventional AI applications from vision to audio to a little bit even to some of the language models right at such tiny scale.
• 5:56
  SPEAKER_00: But then if you have this chip, right, and you can run there, it might suck up a lot of energy.
• 6:02
  SPEAKER_00: So, what's key in kind of this edge AI devices, if you want to use them as effectively as possible, each cur, each bit of milliamps or microamps needs to be directed to where it's actually needed.
• 6:15
  SPEAKER_00: And when you don't need to use a specific component, the best thing is to turn it off.
• 6:20
  SPEAKER_00: So if you want to do this, you must have some sort of support system.
• 6:24
  SPEAKER_00: So when you turn it off and back on, it resumes from where it was stopped, not to restart from the beginning.
• 6:30
  SPEAKER_00: And that's where it comes the um effectiveness of R RAM or MRAM into play.
• 6:35
  SPEAKER_00: We can turn things off very quickly, turn it back on in microseconds, right, granularity, and preserving the system state that's actually orders of magnitude faster than if you want to do the same thing with flash memory devices.
• 6:48
  SPEAKER_00: So this is where the non-volatile memory of MRAM and R RAM becomes really handy.
• 6:52
  SPEAKER_00: So to keeping the state of your processing system, right, where it is, you turn it off, wait for the data to be fed from the sensors, and then you can turn the own system again, you can run your AI workload.
• 7:05
  SPEAKER_00: Once you finish that classification, turn it off, wait for the data to be fed again, and then turn it back on.
• 7:10
  SPEAKER_00: Why we're doing this?
• 7:11
  SPEAKER_00: Because when you interface edge AI devices, the sensors or the environment actually have a very low duty cycle.
• 7:18
  SPEAKER_00: We're talking about hundreds of hertz, kilohertz at maximum.
• 7:21
  SPEAKER_00: So we're talking about inter signal latency of around milliseconds, right?
• 7:27
  SPEAKER_00: So if you can, or tens of milliseconds.
• 7:29
  SPEAKER_00: If you can run things in microseconds and then you can turn the system off, then you would save a lot of power than making the entire system going on or even in a retainable data state for all for all its own operation.
• 7:42
  SPEAKER_00: So as a result of doing all this, we achieve up to 12 trillion operations per watt efficiency, and we have fabricated this the chip here with a 22 nanometer technology node.
• 7:52
  SPEAKER_00: That's what we have done yesterday.
• 7:53
  SPEAKER_00: Today we're doing it with 16, and tomorrow we're going even for more advanced technology nodes.
• 7:59
  SPEAKER_00: And when you're doing this at uh um full force operation, 30 billion operations per second at two milliwatts active power consumption.
• 8:10
  SPEAKER_00: We have taken this chip and then we start benchmarking it right using the MLR ML perf uh uh benchmark suite, and we found out that compared to the absolute best, right, publicly available results in the uh this kind of category, we achieve orders of magnitude improvement in energy consumption.
• 8:30
  SPEAKER_00: So we performance-wise, we achieve as fast as the fastest or even slightly faster, but our key selling point here is that we can run things and like mean 20 times or even 200 times lower energy than what's possible with other vendors.
• 8:46
  SPEAKER_00: So this makes us a natural fit for everything that runs on a battery, interfaces with natural signals, and you works on load-duty cycles.
• 8:55
  SPEAKER_00: So from hearables to wearables, right, to um IoT sensor nodes in industrial IoT in home automation, in AR glasses, and even in drones, right?
• 9:04
  SPEAKER_00: Where you can use them not just in enhancing the functionality of drones, but even have core impact on the drone control systems.
• 9:14
  SPEAKER_00: All right, so that comes at the chip level and the application level, but then you want to be able to program this chip and you want to have um the ability to program it as efficiently as possible.
• 9:26
  SPEAKER_00: So all of our SDK is actually compatible with existing AI applications, right?
• 9:33
  SPEAKER_00: Or AI suites.
• 9:34
  SPEAKER_00: So TensorFlow or PyTorch, if you have your AI uh application developed there, it's very easy, almost as easy as a click of a button to get it ported into our chip.
• 9:44
  SPEAKER_00: Not just that, the SDK even allows you to access the chip remotely.
• 9:48
  SPEAKER_00: So let's say today, right, if you have an application with you in C, you can head to our booth, right?
• 9:56
  SPEAKER_00: You can run it onto our chips that's available in Singapore.
• 9:59
  SPEAKER_00: So we have the boards in Singapore, you can remotely access it here from Taiwan or pretty much anywhere in the world.
• 10:05
  SPEAKER_00: That allows target customers, if they have a ready application and they want to have a quick and dirty way of evaluating it, right, to say, okay, if this chip is a good fit or not, you can just go online, run your application on our chip, not a simulated infrastructure, and you will get the results in real time.
• 10:21
  SPEAKER_00: So you will know actually if it works or not.
• 10:24
  SPEAKER_00: Then that reduces the time needed for development, right?
• 10:27
  SPEAKER_00: Only you would get the chip if you want to interface it with certain sensors or if you want to start your design ins, right, on your custom board.
• 10:35
  SPEAKER_00: We support both bare metal and Zephyr operating system, right, that runs on the chip.
• 10:40
  SPEAKER_00: So if you have certain routines running on Python, we can still make it run for you.
• 10:44
  SPEAKER_00: If you want to run it bare metal for more optimized gains, you still have that support for you as well.
• 10:50
  SPEAKER_00: But again, right, running alone, right, things we can get as best as we can, right?
• 10:56
  SPEAKER_00: The best thing that you will get always is through collaboration.
• 10:59
  SPEAKER_00: And we've been very fortunate, right, and very um honored to have collaborated with literal labs, right, that they develop highly optimized uh versions of any kind of AI workloads.
• 11:11
  SPEAKER_00: And we work together, right, to try to see if their optimizations can work into our chip.
• 11:16
  SPEAKER_00: And the answer is yes, and actually we achieve significant amount of speed-ups and simultaneous inner reduction at the same time, compared to existing state-of-the-art, right, widely adopted edge AI chips, right, simultaneously, hundred times faster and six hundred times or even a thousand times reduction in the amount of energy consumed.
• 11:39
  SPEAKER_00: Right, this is significant.
• 11:40
  SPEAKER_00: And that comes from the not just our chip, not just from literal labs, but this kind of close collaboration together.
• 11:47
  SPEAKER_00: And as we combine these efforts, right, at the same time, we are looking more more and more gains, not just that, but even looking for more sophisticated applications to be ported into these kind of platforms with smaller and smaller footprint.
• 12:03
  SPEAKER_00: In addition to this, right, we also had the honor to, collaboration with Addo Electronics, to design modular boards, right?
• 12:12
  SPEAKER_00: You have the chip, you have the software, right, but you want the ability to very quickly, right, develop a platform for certain kind of use case applications.
• 12:22
  SPEAKER_00: So you can actually check it in our booth when we have this very small board, it's actually four centimeter by two centimeter, right?
• 12:29
  SPEAKER_00: And then you can click, right, certain sensors board on top of that.
• 12:33
  SPEAKER_00: So we have a module for predictive maintenance, we have a module for uh biometric signals, we have a module for cold asset tracking, and suddenly this board is not just an AI evaluation board, it becomes a sensor node.
• 12:47
  SPEAKER_00: And you can stack multiple of this on top to have different kinds of functionality for wireless communication, extended storage, more sensors, right, that can become in a 3D kind of fashion.
• 12:59
  SPEAKER_00: So this one will be available also for sale, right, through Aero Electronics and even interested partners who want to work for uh further with us and have their own customized set of sensor boards as well.
• 13:12
  SPEAKER_00: So that took us from the application from the sorry from the chip to the module, and now to the applications where we can use many of these kind of combined efforts to achieve significant energy efficiency.
• 13:25
  SPEAKER_00: And one key example that I wanted just to shed the highlights on before I wrap my talk is the use of AI in drones.
• 13:34
  SPEAKER_00: And when you go and like mean if you try to think about, okay, how can I enhance the functionality of drones, you start thinking about, okay, let's make it vision, more localization, and so on.
• 13:43
  SPEAKER_00: But if you go to a drone company, they will tell you you can have the best electronics in the world.
• 13:48
  SPEAKER_00: If you don't solve the battery issue, which has been consumed majorly by the mechanical components, right, anything that you do in the electronics is a nice thing to have.
• 13:58
  SPEAKER_00: It's not a core challenge for them.
• 14:00
  SPEAKER_00: So we took this, so so we took their words and we started thinking if we can use our chip, right, combined with AI applications for an AI control to enhance, right, the battery subsystem, increase the flight endurance, which means that for the same charge, can the drone stay in the air for a longer time?
• 14:20
  SPEAKER_00: So we started the modeling, we started the simulation with hardware in the loop, and we did an extensive simulation, and we have the answer was yes, you can achieve actually up to 60%, meaning that a flight that could take one hour on a single battery charge with our solution, with our chip mounted on it, can now take one hour and a half.
• 14:40
  SPEAKER_00: Or you can simply reduce the battery lifetime for increased payload uh capacity.
• 14:44
  SPEAKER_00: So we did this not just on one drone type, we did it on three drone types, and we did a lot of trials on it up to 300 times or more, and we can see a consistent right increase of the flight endurance that reaching 70% or 80%.
• 14:59
  SPEAKER_00: And that actually got a lot of um drone manufacturers to be interested, and we are working in close cooperation with them now to pour this solution into even open source hardware platforms and even higher grade ones, like to talk about like 25-pound uh kind of drones in this case.
• 15:17
  SPEAKER_00: So, again, this is just an example of what we can do in the chip and the idea that when you look at the entire innovation from end to end, you find all of these hidden knobs, and when you tweak all of them, you start seeing a lot of gains that unlocks things that was not possible before.
• 15:33
  SPEAKER_00: And with that, I would like to close my talk.
• 15:35
  SPEAKER_00: Thank you so much for listening.