Latest / EDGE AI POD / How Edge AI Moves From Cloud Hype To Real-World Impact

Transcript

How Edge AI Moves From Cloud Hype To Real-World Impact · EDGE AI POD

• 0:24
  SPEAKER_01: Generative AI, is it cool or what?
• 0:28
  SPEAKER_01: Technologies are cool in my mind.
• 0:31
  SPEAKER_01: The problem with technologies, with new technologies recently is that we just got used to the fact that we have a new AI model, new AI capability almost every week.
• 0:45
  SPEAKER_01: So we sometimes miss the big picture.
• 0:48
  SPEAKER_01: The big picture is that new technologies are not always just faster, better computers.
• 0:57
  SPEAKER_01: They really reshape everything that we're doing.
• 1:01
  SPEAKER_01: So let me take you to the first industrial revolution.
• 1:06
  SPEAKER_01: So we managed to use steam.
• 1:10
  SPEAKER_01: So what?
• 1:10
  SPEAKER_01: It's not just a better machine.
• 1:12
  SPEAKER_01: Okay?
• 1:13
  SPEAKER_01: The steam engine has really made the world smaller with steamboats, steam trains that could transfer people and goods all around the world.
• 1:25
  SPEAKER_01: Now the next generation brought us electricity.
• 1:29
  SPEAKER_01: So now we can also work during night because there is light.
• 1:34
  SPEAKER_01: We can work during winter because there's heating.
• 1:39
  SPEAKER_01: So it's a huge change to our lifestyle, but it even changed our biology.
• 1:45
  SPEAKER_01: Think about that.
• 1:46
  SPEAKER_01: Are you taller than your parents?
• 1:49
  SPEAKER_01: Are they taller than their parents?
• 1:52
  SPEAKER_01: Is the world getting taller?
• 1:54
  SPEAKER_01: What do you think?
• 1:57
  SPEAKER_01: Yes, I mean you don't think about that when you invent electricity, but then you have refrigerators, then you have better food, better health care, and look at that.
• 2:08
  SPEAKER_01: This is the average height of men in Germany.
• 2:11
  SPEAKER_01: They gained like 20 centimeters.
• 2:14
  SPEAKER_01: It's a lot, right?
• 2:15
  SPEAKER_01: It's think about that.
• 2:16
  SPEAKER_01: Too bad they kept me so short, but it's it's pretty amazing.
• 2:21
  SPEAKER_01: And then came computers.
• 2:23
  SPEAKER_01: I don't have to tell you how much this changed everything.
• 2:26
  SPEAKER_01: The way we behave, the way we communicate with each other, everything.
• 2:30
  SPEAKER_01: And the next revolutions are gonna make bigger changes.
• 2:36
  SPEAKER_01: Okay, this is not a linear growth.
• 2:40
  SPEAKER_01: This is um an exponential, exponential revolution.
• 2:45
  SPEAKER_01: Okay, and I think what enabled AI eventually is the fact that we taught the machine to speak our language.
• 2:54
  SPEAKER_01: Okay, you and I have uh language barriers, but computers can understand everything from us, right?
• 3:02
  SPEAKER_01: The minute we were able to represent our language using tokenization, put it in embedding space, attention, and so on, and train large language models, we basically opened the door for computers to all of our knowledge.
• 3:22
  SPEAKER_01: I mean, computers can generate images and videos and audio, it's it's really nice, but text is more important because after all, most of our knowledge is text.
• 3:34
  SPEAKER_01: And then we taught the machines, we taught computers to generate text, right?
• 3:40
  SPEAKER_01: So we can interact with them, we can chat with them, and now we can measure their intelligence.
• 3:46
  SPEAKER_01: Are they smarter than us?
• 3:48
  SPEAKER_01: What do you think?
• 3:50
  SPEAKER_01: Is computer already smarter than us?
• 3:53
  SPEAKER_01: So the average IQ of a person is about 100.
• 3:58
  SPEAKER_01: I want to believe that most of the people in this room are on the right side of this curve, but so as most of the latest models.
• 4:08
  SPEAKER_01: Okay, so yes, AI is already smarter than us.
• 4:12
  SPEAKER_01: We can argue about the details, how do you measure IQ and stuff like that, but in any possible test, they are better.
• 4:20
  SPEAKER_01: In IQ, in EQ, they're even more funny than us.
• 4:24
  SPEAKER_01: Okay?
• 4:24
  SPEAKER_01: That's really insulting.
• 4:26
  SPEAKER_01: But for me, the shocking fact is how fast this has all happened, right?
• 4:31
  SPEAKER_01: It took AI one year, one year, to move from an average person with an IQ of 100 to a genius with an IQ of 140.
• 4:42
  SPEAKER_01: Okay, that's pretty fast.
• 4:43
  SPEAKER_01: Now, what is next?
• 4:45
  SPEAKER_01: I mean, it when we're moving so fast, it's very difficult to understand where technology is heading, what comes next.
• 4:54
  SPEAKER_01: So if we zoom out a little bit, I think we can see the pattern, okay?
• 5:02
  SPEAKER_01: And there is a pattern because technology evolved in, in my mind, in predictable directions.
• 5:09
  SPEAKER_01: Okay, so let's take this device, sickle, right?
• 5:15
  SPEAKER_01: One blade, and I believe that when people stopped uh picking up crop with their their hands and start using blades, it was a huge technology advancement, right?
• 5:24
  SPEAKER_01: But then somebody said, okay, using the same stroke, I can use two blades, right?
• 5:30
  SPEAKER_01: So multiple blades will be better, will be more efficient, so why not?
• 5:35
  SPEAKER_01: Now, if this machine can be mobile, I can cover more ground.
• 5:40
  SPEAKER_01: If this machine is autonomous, then I'm free up to do other stuff.
• 5:45
  SPEAKER_01: And if I have one machine like that that is autonomous, why not having many of them working like a swarm, right?
• 5:53
  SPEAKER_01: And if one of them is getting you know broken, they can cover for me for each other.
• 5:58
  SPEAKER_01: This is nice.
• 5:59
  SPEAKER_01: So this is technology trends, and you can see them in many domains, not only in agriculture.
• 6:06
  SPEAKER_01: You know, here you can see fishing, you can see it in many places.
• 6:10
  SPEAKER_01: Recently, unfortunately, we also see that in military technologies, right?
• 6:16
  SPEAKER_01: You have one rocket, multiple missiles, or one missile with multiple heads, and then mobile launchers, and then guided missiles, you know, guided by GPS, by vision, and then a swarm.
• 6:30
  SPEAKER_01: In all of these technologies, in all of these examples, parallelism means efficiency, mobility means opportunities to cover more.
• 6:43
  SPEAKER_01: If it's autonomous, I'm free up to do other stuff.
• 6:46
  SPEAKER_01: And if it's distributed like a swarm, it opens up new opportunities and new advantages like redundancy, scalability, and more.
• 6:58
  SPEAKER_01: And AI is not different.
• 7:01
  SPEAKER_01: Last year's we've tried to improve AI by adding more and more layers to the models.
• 7:08
  SPEAKER_01: Okay, so more parallelism, more parallelism to our GPUs, and this is how we made it.
• 7:14
  SPEAKER_01: Then we tried to make it mobile, right?
• 7:17
  SPEAKER_01: We want to take these models and put them in more devices.
• 7:22
  SPEAKER_01: Okay, so how do you do that?
• 7:24
  SPEAKER_01: You invent all kinds of protocols like MCP, so it would be easier to put them in many places and communicate with them.
• 7:32
  SPEAKER_01: You try to come up with smaller language models, you're coming up with better, more efficient accelerators like NPUs, the neural processing unit, or you come up with better algorithms like speculative decoding.
• 7:47
  SPEAKER_01: Feel free to take a look at my video explaining exactly that.
• 7:51
  SPEAKER_01: What is the next step?
• 7:53
  SPEAKER_01: Autonomous.
• 7:55
  SPEAKER_01: Right?
• 7:55
  SPEAKER_01: We all heard about AI agent.
• 7:58
  SPEAKER_01: What is an AI agent?
• 7:59
  SPEAKER_01: It's basically an AI-powered system or application, right?
• 8:05
  SPEAKER_01: That is autonomous.
• 8:06
  SPEAKER_01: It can consume data and make decisions for us without the constant need for input, right?
• 8:13
  SPEAKER_01: So it's autonomous.
• 8:15
  SPEAKER_01: And then the question is what comes next?
• 8:17
  SPEAKER_01: What comes next?
• 8:19
  SPEAKER_01: The edge.
• 8:20
  SPEAKER_01: Distributed, autonomous, AI systems everywhere.
• 8:26
  SPEAKER_01: This is just logical, you know, trend that will bring us exactly there.
• 8:33
  SPEAKER_01: We want to take the intelligence, we want to take the wisdom of AI and distribute it and put it closer to human, closer to the edge, closer to where the data is.
• 8:46
  SPEAKER_01: This is exactly what we're trying to do.
• 8:48
  SPEAKER_01: And you're gonna see that anywhere.
• 8:52
  SPEAKER_01: So an LLM, a large language model running in the cloud is great, but I want it to run on my laptop.
• 9:00
  SPEAKER_01: Okay?
• 9:00
  SPEAKER_01: I don't need generic answers and advices.
• 9:05
  SPEAKER_01: I want something, I want advices and answers customized to my style, to my you know intelligence, to my needs, and only a system that is familiar with with what I'm doing, okay, that have access to all my communication, documentation, mail, etc., can do that.
• 9:30
  SPEAKER_01: We want to bring AI closer to the data.
• 9:34
  SPEAKER_01: The foundation of AI is data.
• 9:37
  SPEAKER_01: Where is that data being generated?
• 9:39
  SPEAKER_01: Not in the cloud, right?
• 9:40
  SPEAKER_01: The data is being generated in a factory.
• 9:43
  SPEAKER_01: So a robot in a factory can detect right away that this part has a defect and it has to be discarded.
• 9:52
  SPEAKER_01: It doesn't need to go to the cloud for that, right?
• 9:56
  SPEAKER_01: You need to bring the wisdom to the edge, to where things are happening.
• 10:01
  SPEAKER_01: And if you have a smart city camera, it doesn't need to stream all the video to a command center.
• 10:09
  SPEAKER_01: It can detect by itself that has accidents that just occurred, right?
• 10:14
  SPEAKER_01: It can alert the police, the hospital, it can, you know, collaborate with other intersections to create a green wave for for the ambulance, and so on.
• 10:26
  SPEAKER_01: And I haven't even started to talk about humanoids, right?
• 10:30
  SPEAKER_01: They're gonna be distributed, they're gonna be mobile, they're gonna be autonomous.
• 10:40
  SPEAKER_01: Exactly this.
• 10:42
  SPEAKER_01: The question is, why is this not happening?
• 10:45
  SPEAKER_01: Okay, we have millions or billions of devices waiting to be to get smarter, and we got lots of AI models want to be deployed.
• 10:56
  SPEAKER_01: Okay, so what is preventing us from getting there?
• 10:59
  SPEAKER_01: There are many problems, but just to mention a few.
• 11:03
  SPEAKER_01: The first problem is hardware.
• 11:05
  SPEAKER_01: Okay, edge devices have unique hardware requirements.
• 11:10
  SPEAKER_01: Okay, when you're working in the cloud, you theoretically can have endless compute power, endless memory space, right?
• 11:17
  SPEAKER_01: And this is not the case in the real world.
• 11:21
  SPEAKER_01: Edge devices has limited amount of power.
• 11:26
  SPEAKER_01: They can dissipate limited amount of compute power, limited memory, and so on.
• 11:31
  SPEAKER_01: If you want to bring AI capabilities to the edge, you want to have specialized machines, specialized hardware for that.
• 11:39
  SPEAKER_01: So, just like our brain, think about our brain.
• 11:42
  SPEAKER_01: Our brain is an excellent edge device, uh, example for an edge device, right?
• 11:48
  SPEAKER_01: It can dissipate around 20 watts, so low power.
• 11:52
  SPEAKER_01: It has unique regions to decipher, to process vision, unique special regions to process audio, special specialized regions to take decisions, and so on.
• 12:09
  SPEAKER_01: Okay?
• 12:09
  SPEAKER_01: Now look at the latest um microprocessor from Intel.
• 12:15
  SPEAKER_01: This is codenamed the Ultra Core III, codenamed Panther Lake.
• 12:20
  SPEAKER_01: It has a CPU, it has a GPU, it has an NPU, lots of acceleration.
• 12:25
  SPEAKER_01: It's a highly heterogeneous system, and we need all of that for the future edge devices, right?
• 12:31
  SPEAKER_01: Think about robots that need to process multiple cameras to construct a 3D view of the world around him and navigate through it just like we humans do.
• 12:44
  SPEAKER_01: So you need specialized hardware, and you need here, you got here uh CPUs with multiple cores, right?
• 12:51
  SPEAKER_01: Efficient core, performance core.
• 12:53
  SPEAKER_01: You got here image processing and media processing cap acceleration inside.
• 12:58
  SPEAKER_01: You got here uh very large GPU and an NPU.
• 13:02
  SPEAKER_01: The problem is, and this brings me to the problem number two, that it introduces software problems, okay?
• 13:10
  SPEAKER_01: I mean, we all know that programming a CPU, a GPU, or an NPU is totally different.
• 13:16
  SPEAKER_01: Which makes sense, right?
• 13:17
  SPEAKER_01: It's a different architecture, different instruction set, different memory mapping, and so on.
• 13:24
  SPEAKER_01: Okay, so this is problem number two.
• 13:27
  SPEAKER_01: And for this, we have introduced what we call open-edge platform.
• 13:32
  SPEAKER_01: It's an open source, free-to-use bundle of software that basically have everything you need in order to get AI into edge devices.
• 13:43
  SPEAKER_01: So it starts with the lower level manageability and operating system and everything, but it goes to what we call edge AI libraries, like in with tools like OpenVino, if you want to use um highly efficient uh inference on CPU, GPU, NPU.
• 14:04
  SPEAKER_01: By the way, it's open source, it runs also on ARM, on Raspberry Pi, whatever you want.
• 14:09
  SPEAKER_01: If you want to process full video pipelines, we have the deep learning streamer.
• 14:14
  SPEAKER_01: If you want to take multiple cameras into constructing your 3D world, okay, and understand how it operates, use Intel Scenescape.
• 14:25
  SPEAKER_01: We have a nice demo over there.
• 14:27
  SPEAKER_01: Just go to the Dell booth and take a look.
• 14:30
  SPEAKER_01: How we take multiple cameras and just track real objects in the world and Getty.
• 14:36
  SPEAKER_01: Getty is for in my mind is a great tool that can help people train models, custom models, without the need to be an expert, right?
• 14:48
  SPEAKER_01: It's all GUI.
• 14:49
  SPEAKER_01: You get in your data, annotate it really quickly, and you get a model which you can use in the cloud or on any edge device.
• 14:59
  SPEAKER_01: So try it out.
• 15:00
  SPEAKER_01: There's many other tools, of course.
• 15:02
  SPEAKER_01: And there is an AJI suite on top with ready-to-use solutions for multiple use cases for retail manufacturing, for smart city, for robotics.
• 15:15
  SPEAKER_01: It's just if you look there, I'm pretty sure you're gonna find uh a reference design, an example, a sample code you can use right away, copy, learn how to use it.
• 15:27
  SPEAKER_01: So this is the open edge platform.
• 15:30
  SPEAKER_01: I think the hardware solutions with the right software solutions will bring us to the right place we're looking for, but AI still has some mileage to go, and I just want to talk in two, three more minutes about how do I see the next trends in AI, what's going next.
• 15:54
  SPEAKER_01: So in the near future, we're gonna see models getting more nimble, lighter, getting more cognitive, more autonomous, and more personal.
• 16:07
  SPEAKER_01: And I'll try to elaborate on that.
• 16:09
  SPEAKER_01: So right now we see how models are progressing.
• 16:14
  SPEAKER_01: They are getting smarter and smarter, but they're also getting bigger and bigger.
• 16:18
  SPEAKER_01: They're getting getting too big.
• 16:20
  SPEAKER_01: Okay, and if you look inside them, right, some of them are really sparse.
• 16:25
  SPEAKER_01: I don't want to say they're empty, but they're really not very efficient.
• 16:30
  SPEAKER_01: Okay, and you know, we can apply all kinds of um algorithms on large models, quantization.
• 16:39
  SPEAKER_01: We're quantizing now to 4-bit, and we hardly see any accuracy degradation.
• 16:45
  SPEAKER_01: We're running pruning, distillation, and so on.
• 16:50
  SPEAKER_01: So the potential is there, and we're gonna see more and more smaller models.
• 16:56
  SPEAKER_01: We're gonna see much more targeted models, okay?
• 17:00
  SPEAKER_01: Right now, what you're doing is you're getting a model that knows everything about everything, right?
• 17:07
  SPEAKER_01: But I'm only wanna ask about a medical question, right?
• 17:11
  SPEAKER_01: So when you want to read about, I don't know, dinosaur or something, you don't read the whole encyclopedia, right?
• 17:18
  SPEAKER_01: So we're gonna see more and more targeted, focused models for doctors, for lawyers, I don't know, for engineers.
• 17:28
  SPEAKER_01: They're gonna be smaller, they're gonna be smarter, okay, and they're gonna be focused to exactly what I need.
• 17:35
  SPEAKER_01: We're gonna see more retriever-based, okay?
• 17:38
  SPEAKER_01: Why hold all the information inside the model when I can just go and look it up using you know techniques like RAG in the most updated real-time resources, right?
• 17:53
  SPEAKER_01: You don't have to hold everything in in your in your belly, in your mind, in your brain, in the model.
• 18:00
  SPEAKER_01: So we're gonna see much many more retrieval-based, and we're gonna see much more on fine-tuning.
• 18:08
  SPEAKER_01: Today, when you get a model, this is a model, okay?
• 18:11
  SPEAKER_01: But I'm pretty sure in the near future we're gonna see lots of models getting improved over time.
• 18:18
  SPEAKER_01: It doesn't have to be fast, you don't need a big machine for that.
• 18:21
  SPEAKER_01: It can take 10 hours.
• 18:23
  SPEAKER_01: I don't care.
• 18:24
  SPEAKER_01: Overnight, just fine-tune the model.
• 18:27
  SPEAKER_01: And then my smart surveillance camera knows that this strange person who stopped, you know, who walked by every every day at 10 o'clock is the mailman, right?
• 18:39
  SPEAKER_01: It's not somebody that I should suspect, right?
• 18:43
  SPEAKER_01: Or think about two cars, two Tesla cars that just left the factory in Texas.
• 18:51
  SPEAKER_01: Okay, one of them was shipped to India and the other was shipped to Australia.
• 18:56
  SPEAKER_01: Okay, and these two cars are driving every day, taking millions of photos, right?
• 19:02
  SPEAKER_01: And at night they plug it in, upload everything, retrain, get a new model.
• 19:08
  SPEAKER_01: So after a month, the car in India will know how to avoid cows on the road, and the car in Australia will be careful of kangaroos, for example.
• 19:20
  SPEAKER_01: Okay, but the models will be local, will be you know, ongoing, um improving themselves as time goes by.
• 19:30
  SPEAKER_01: Okay, models are gonna be more cognitive.
• 19:33
  SPEAKER_01: We see this already happened with models combining text and vision and others, they will be more adaptive, more aware to the context and to the situation, and we already see models that can explain themselves.
• 19:49
  SPEAKER_01: And models are gonna be much more personal, okay?
• 19:52
  SPEAKER_01: Think about there are already um services like that, paid services like that, but think about that.
• 19:59
  SPEAKER_01: You have a student who wants to solve an equation.
• 20:03
  SPEAKER_01: I don't need the my model to just tell him what the answer is, right?
• 20:08
  SPEAKER_01: I need the model to take him hand by hand.
• 20:11
  SPEAKER_01: So I want the model to ask me, do you know what is the equation?
• 20:15
  SPEAKER_01: No, let me tell you.
• 20:16
  SPEAKER_01: Do you know what is a differential equation?
• 20:18
  SPEAKER_01: Let me show you.
• 20:19
  SPEAKER_01: So just take me the path to understanding so I can really solve it the next time, right?
• 20:25
  SPEAKER_01: I don't need the answer.
• 20:26
  SPEAKER_01: And for this, you gotta have a system that actually knows me, right?
• 20:32
  SPEAKER_01: Not giving me some generic answer.
• 20:35
  SPEAKER_01: Here is a great example of personal use by Microsoft CEO, right?
• 20:41
  SPEAKER_01: And you can see how he's asking AI to prepare him to the next meeting based on all his interaction with this person.
• 20:52
  SPEAKER_01: In order to do that, the system really has to know him, right?
• 20:56
  SPEAKER_01: It needs to have access to all of his emails, to all of his communication with this person, right?
• 21:04
  SPEAKER_01: And read maybe all his presentation, all his um whatever he wrote, right?
• 21:11
  SPEAKER_01: This way, you know, the system can really give you something very targeted, very stitched to your need, okay?
• 21:20
  SPEAKER_01: But again, it has to be on the edge, it has to know what are your needs, what is your style, and so on.
• 21:26
  SPEAKER_01: And it can customize to my specific special needs.
• 21:30
  SPEAKER_01: For example, if I have a dyslexia, just just the thought, right?
• 21:34
  SPEAKER_01: It can present me the text in the way that it's easier for me to read.
• 21:39
  SPEAKER_01: Just one example, and you can think about so many of these such customization, and it's gonna be autonomous.
• 21:47
  SPEAKER_01: In the near future, AI models or agents will work for us, will find for us the next vacation, will read, you know, will schedule meetings for us, etc.
• 22:02
  SPEAKER_01: Until then, we need to be, you know, before it happens, we need to trust them.
• 22:08
  SPEAKER_01: And right now, I don't feel so comfortable giving an AI agent the keys to my bank account or to my stock account to do whatever he wants.
• 22:20
  SPEAKER_01: Okay, it will take us some time to really trust them.
• 22:23
  SPEAKER_01: So, but it will happen.
• 22:25
  SPEAKER_01: And until then, until then, feel free to find me on LinkedIn.
• 22:32
  SPEAKER_01: Feel free to find me on YouTube.
• 22:34
  SPEAKER_01: I have a YouTube channel called AI with Guy, um, where I post like very small short videos about you know AI and and stuff.
• 22:46
  SPEAKER_01: And feel free to use the Open Edge platform.
• 22:48
  SPEAKER_01: We built it for you.
• 22:50
  SPEAKER_01: So enjoy it and thank you.