Latest / EDGE AI POD / Transforming Human-Computer Interaction with OpenVINO

Transcript

Transforming Human-Computer Interaction with OpenVINO · EDGE AI POD

• 0:06
  Speaker 1: So welcome to Raymond Lowe, Intel.
• 0:09
  Speaker 1: He's going to speak about building your chatbots with OpenVINO and AIPC.
• 0:18
  Speaker 1: Raymond, thank you Welcome.
• 0:21
  Speaker 1: Thanks for being with us today, and you have 25 minutes plus five minutes Q&A and the floor is yours.
• 0:33
  Speaker 1: Thanks so much.
• 0:34
  Speaker 2: Perfect.
• 0:34
  Speaker 2: Thank you again.
• 0:35
  Speaker 2: I will have a lot of slides, so bear with me with the slides today.
• 0:39
  Speaker 2: My name is Raymond, I'm from Intel and I've been with Intel for the last five years almost five years so if you're interested in what we're doing, follow me or our team on LinkedIn.
• 0:51
  Speaker 2: You'll see a lot of news about what we're doing at Intel.
• 0:54
  Speaker 2: So oftentimes you'll see a post from me before there's an announcement somehow.
• 1:00
  Speaker 2: So, for example, I work on OpenVINO a lot, so a lot of pre-release materials.
• 1:05
  Speaker 2: Sometimes I test it out and you can see it, but to no further ado.
• 1:08
  Speaker 2: Let's talk about chatbot.
• 1:10
  Speaker 2: People call it agentic sometimes.
• 1:13
  Speaker 2: For me it's really about how can we talk to a computer, and today we'll talk about that, because having a machine talking to you was not easy, and when it does, it's very powerful.
• 1:26
  Speaker 2: So let's think about journey.
• 1:30
  Speaker 2: As yourself, I've done a lot of different programming exercise since I was young, so but one thing really intrigued me today is we can really have a good interactive experience with computers.
• 1:44
  Speaker 2: I remember the first time when I get a computer.
• 1:46
  Speaker 2: You get a mouse and keyboard, you click on it, type some document.
• 1:49
  Speaker 2: It feels good, but what if we can have a proper conversation and what that brings us.
• 1:54
  Speaker 2: An experience like going to the hospital, for example, how many times you walked in.
• 1:59
  Speaker 2: You're trying to figure out, okay, who should I talk to, Where's the nurse right?
• 2:03
  Speaker 2: And oftentimes you go in there the nurse may ask you, uh, why are you here?
• 2:08
  Speaker 2: And then of course we're sick, of course we are not feeling comfortable at that time.
• 2:13
  Speaker 2: I think if we can reduce some of these wait time, I think there's a lot of huge benefit from building such a chatbot, or assistance we call it sometimes and my brother's the doctor right, we saw a lot in inefficiency in the system, like especially for nook ticking and things like that.
• 2:31
  Speaker 2: So, as we see today, this is becoming a reality.
• 2:36
  Speaker 2: I see, actually, when I see the doctor last week or a couple of weeks ago I walk into the office, I saw the person already turn on the cell phone on the edge, recording our conversation, transcribing and making a better experience when things are needed to pay attention to.
• 2:54
  Speaker 2: I think to us at Intel, this is a great opportunity because your PC, while sitting in front of you, can be the best companion, and why I think so is they're not just talking right, the chatbot.
• 3:09
  Speaker 2: Of course you can do large language model.
• 3:10
  Speaker 2: When I first started I started Lama to was like almost ancient time but it was not answering great questions.
• 3:16
  Speaker 2: Sometimes it was like off track elucidating.
• 3:19
  Speaker 2: But today, if you look at what Microsoft released for example 5.3.5, they can answer better question, even with images.
• 3:28
  Speaker 2: So imagine now you put a question about like what's happening?
• 3:32
  Speaker 2: To my eye they actually can do better and we saw just last week I was at Microsoft Build.
• 3:38
  Speaker 2: We saw scientists are using these tools for discovery and more and more.
• 3:42
  Speaker 2: And that's why I think about chatbot is just really like how can we have a better way to communicate with computer in a deeper way?
• 3:49
  Speaker 2: Right, Think about even doing research and thought processes.
• 3:53
  Speaker 2: And for consumer I don't know if anyone have babies here.
• 3:57
  Speaker 2: I do have a baby and it's one year old.
• 3:59
  Speaker 2: And the number one question I get asked by my wife a lot is like what's my baby doing?
• 4:05
  Speaker 2: And imagine we can have smart camera that you can set up without coding.
• 4:09
  Speaker 2: All you need is put like a large language model behind it, like with a visual language model, and then you can have questions like oh, my baby was trying to play with the computer, play with the keyboard, and that's something actually we're seeing on the edge.
• 4:21
  Speaker 2: It's like, hey, we can actually make extremely smart computer without a lot of computation power things.
• 4:27
  Speaker 2: It may not be very accurate, but we're just at the beginning, right.
• 4:31
  Speaker 2: And then how much more we can get within a year, that's something worth thinking about.
• 4:36
  Speaker 2: And one thing I think a lot of us like when we build a lot of systems on the edge is you never have that big enough model.
• 4:44
  Speaker 2: You always have a small model that actually can fit into the memory.
• 4:48
  Speaker 2: You may have like maybe more limited reasoning.
• 4:51
  Speaker 2: So we also see the next generation is more like agentic with Bragg on top.
• 4:56
  Speaker 2: So why that's important is we test a lot of these.
• 4:59
  Speaker 2: For example, in this case with Lama free A billion.
• 5:03
  Speaker 2: It works extremely well for, let's say, simple use cases and when you have rack which allow you to, uh, augmenting information to the chatbot.
• 5:12
  Speaker 2: Let's example, ask what's AIPC here.
• 5:15
  Speaker 2: It will not know it if I didn't give you the document right.
• 5:20
  Speaker 2: So it's important in the edge use case.
• 5:23
  Speaker 2: Maybe sometimes you want to provide extra information.
• 5:25
  Speaker 2: So then, um, it actually be contextual aware.
• 5:28
  Speaker 2: Think about you in the restaurant uh, you want to build something that, uh, with your burger menu, and we can do that now.
• 5:34
  Speaker 2: We can do it extremely well with rack and also some agentic checkbox, to know to the very final piece.
• 5:42
  Speaker 2: Um, when we think about computer intel, we've been supporting a lot of different models.
• 5:47
  Speaker 2: And then again I talked about OpenVINO.
• 5:49
  Speaker 2: I haven't introduced it properly, but what we notice is we accelerate some of the work so well that only a couple of years ago or a year or two ago, these are not even possible locally.
• 6:01
  Speaker 2: So what you're seeing on this computer right now, which I capture on my own machine, I'm running one of the image generation on the local GPU, the iGPU.
• 6:11
  Speaker 2: I can generate this beautiful teddy bear in about eight seconds.
• 6:15
  Speaker 2: Eight seconds like, finally, like a beautiful teddy bear.
• 6:21
  Speaker 2: That's faster than any human drawing ever and the quality is as good as some of the photography I took myself.
• 6:26
  Speaker 2: So that's very intriguing.
• 6:27
  Speaker 2: And the same source code.
• 6:29
  Speaker 2: If you upgrade your GPU.
• 6:30
  Speaker 2: I don't you ever follow the Intel Arc tried, I am.
• 6:35
  Speaker 2: I'm the fan boy.
• 6:36
  Speaker 2: That thing is a beast.
• 6:39
  Speaker 2: It has 16 gig of RAM.
• 6:40
  Speaker 2: It actually like a couple years, years ago when it released actually, um, we, you were still.
• 6:44
  Speaker 2: We start building AI examples on it and then we realized they're super powerful for the price point we're giving.
• 6:50
  Speaker 2: So, um, that's a great example how we see um programming like a chatbot or creating um very interactive material on the edge, and the GPU you have today on your laptop was like insufficient, and in this case, have today on your laptop will certainly be sufficient, and in this case we can finish this in two seconds, as you can see, Very fast.
• 7:12
  Speaker 2: So this brings to the very important question is what is happening at Intel and what happened to some of the work we have done?
• 7:20
  Speaker 2: This is very new, by the way.
• 7:22
  Speaker 2: I just released this.
• 7:23
  Speaker 2: Last week.
• 7:24
  Speaker 2: We talked about Microsoft Build.
• 7:25
  Speaker 2: We are now collaborating with many, many, many great customers and partners, and then one thing we collaborate on is a lot of software component that you may not have heard about today until maybe now is Microsoft released something called the AI Foundry or we have the Windows ML.
• 7:44
  Speaker 2: What we have done is actually we have put some of our technology, for example, openpino quantization here you can see it left and then also we profile runtime to the Microsoft ecosystem.
• 7:57
  Speaker 2: So later on, when you buy your PC from, let's say, best Buy or some other, maybe Costco.
• 8:03
  Speaker 2: It will preload OpenPino for you and when you download your model, you will get the best performance out from it.
• 8:10
  Speaker 2: I think this is a life changing for me.
• 8:12
  Speaker 2: When I started Intel four years, five years ago, we joke around it, right.
• 8:16
  Speaker 2: Why don't we have this pre-installed at Windows?
• 8:18
  Speaker 2: Things, not a joke anymore.
• 8:21
  Speaker 2: So it's happening as we move forward with, let's say, the new way of thinking about building software on PC, especially on the AI front.
• 8:30
  Speaker 2: The model will be pre-compressed for you and then we'll have a runtime that's always ready so you get the best performance out from your silicon.
• 8:37
  Speaker 2: So and this is something I spent two to three weeks to get from our engineers so if you pretty wonder how it works under the hood, I spent two to three weeks to get from our engineers.
• 8:47
  Speaker 2: So if you pretty wonder how it works under the hood, I bring it to you.
• 8:50
  Speaker 2: So my background is software engineering, so when I see diagram like that, it makes me happy.
• 8:53
  Speaker 2: First, we take a screenshot of this.
• 8:55
  Speaker 2: So this is under the hood.
• 8:58
  Speaker 2: Come to our webinar.
• 8:58
  Speaker 2: We'll talk about this further, but for now, I think this is a great example how deeply we integrated into some of the work in Windows and others.
• 9:06
  Speaker 2: Of course, our work works on Linux as well, if you're interested in having portability.
• 9:12
  Speaker 2: So, I talked about a lot of benefits.
• 9:15
  Speaker 2: I talked about why we're here, why we're in the edge.
• 9:17
  Speaker 2: But what's OpenVINO?
• 9:19
  Speaker 2: Why are we here talking about this and why do I promote this?
• 9:22
  Speaker 2: So think about all the great example you saw can come from different models, right, we can have PyTorch.
• 9:29
  Speaker 2: We still have some TensorFlow model coming in, for example.
• 9:32
  Speaker 2: Some of the old school gesture recognition stuff was from TensorFlow and we have All.
• 9:38
  Speaker 2: Next, of course, we have even Pedal Pedal from China.
• 9:42
  Speaker 2: All these model come into our Intel silicon.
• 9:44
  Speaker 2: Oftentimes they're not optimized right.
• 9:46
  Speaker 2: So that's what OpenVINO has been doing we're to really compress this model, making sure that it fits into that small size Could be.
• 9:53
  Speaker 2: Your small laptop can be on the cloud.
• 9:55
  Speaker 2: That's what we've been doing.
• 9:57
  Speaker 2: Right, we have a portfolio of hardware from Intel.
• 10:01
  Speaker 2: So if you think about Intel, we don't just build GPU, we don't just build CPU, we build a lot of things from the edge to the cloud, to the client.
• 10:10
  Speaker 2: So when we think about this coverage, what OpenVINO has been doing is we bring this optimization code which, under the hood, is one API, all of these kind of like open source standard work we have done.
• 10:20
  Speaker 2: We distribute across different hardware here.
• 10:22
  Speaker 2: In this case we even have MPU different hardware here.
• 10:24
  Speaker 2: In this case we even have MPU, which is a new silicon uh that we provide for uh, high efficiency, low power on the edge devices and then we started that a long time ago.
• 10:35
  Speaker 2: We call the DPU, we call the bofidius.
• 10:37
  Speaker 2: So at one point if you are following us on Raspberry Pi, people plug in those USB key that has actually our NPU built in and then we can get really good acceleration.
• 10:48
  Speaker 2: I think it was like 10 times faster than the Raspberry Pi only five watt of power.
• 10:53
  Speaker 2: So that was like one of those accelerators we provide.
• 10:55
  Speaker 2: Why this is important, I think, to us, is when you have this cross platform, cross OS, cross architecture and then cross models, we have a very powerful tool that you can view many things or anything.
• 11:09
  Speaker 2: If you can imagine and remember.
• 11:12
  Speaker 2: My thesis is about having communication with the computer so you can have use cases that actually were only dreamed about back in old days become a reality, for example, from computer vision.
• 11:23
  Speaker 2: Computer vision has been very from old school YOLO, which is almost 10 years ago.
• 11:29
  Speaker 2: We optimize it really good to natural language model that we talk about.
• 11:34
  Speaker 2: Maybe back in old days we called it BERT as well.
• 11:36
  Speaker 2: We can have now, above and beyond, with simulation of robots chatbot we talked about we can even do voice generation and recognition very well.
• 11:46
  Speaker 2: And then well I think everyone talked about this we can even do code generation, so you can do coding with it.
• 11:54
  Speaker 2: There's a lot of these possibilities are getting um explore and also be deployed on the edge, as we see every day.
• 12:01
  Speaker 2: So today I lost count how many.
• 12:03
  Speaker 2: Back in old days we say we have 1,000 of these models.
• 12:06
  Speaker 2: Today we don't even know.
• 12:07
  Speaker 2: Well, we have this huge pool of open source and cross-platform models that you never see and a lot of time, I think, we always think about when you have all these examples right.
• 12:20
  Speaker 2: This is an open question to everybody here.
• 12:23
  Speaker 2: Where do I put a computer?
• 12:26
  Speaker 2: Honestly, for the many years I work in either.
• 12:31
  Speaker 2: Before intel, I was doing my own startup.
• 12:33
  Speaker 2: Today, putting the computer right place is never an easy question.
• 12:37
  Speaker 2: It's the hard question enough, never an easy answer to it.
• 12:42
  Speaker 2: I said today we figure out some, some of it because when we built the silicon at Intel which is what I like, my job a lot it's like I see this new silicon came out and then, hey, there's things that we couldn't believe it was possible before.
• 12:53
  Speaker 2: For example, when we do depth estimation or post-estimation, back in old days, when you're trying to do it on the CPU, it was very slow, it's not interactive.
• 13:09
  Speaker 2: Today, when you bundle with the MPU, it can be low power, it's real time, it can be simply just like running behind the scene, like in the background for you or your CPU and GPU arresting happy.
• 13:16
  Speaker 2: So there's like possibility that I didn't know, we didn't know it was such a great idea.
• 13:21
  Speaker 2: And especially background blur.
• 13:22
  Speaker 2: You see, if my background blur, it's amazing for the MPU, because it's not well, you can have four hours meetings at one, because it will use a lot less power, and then your battery will be very happy.
• 13:35
  Speaker 2: And think about this compute.
• 13:36
  Speaker 2: Where do I put that?
• 13:37
  Speaker 2: I think this is where I think OpenFIN will give you that flexibility.
• 13:41
  Speaker 2: You can switch between this model, that, that model, and that's where I think the beauty of what we're doing.
• 13:48
  Speaker 2: So I'll take a pause.
• 13:50
  Speaker 2: I'll read some questions in the q a.
• 13:51
  Speaker 2: Then I'll talk about something really hard.
• 13:53
  Speaker 2: I'm gonna switch my self to the browser or take a look, I think uh also we have ash and touch behind the scene.
• 14:01
  Speaker 2: Actually, ash and tasha has been answering questions.
• 14:02
  Speaker 2: She's from open fino, oh no, oh no, from OpenEdge platform.
• 14:07
  Speaker 2: So we work together.
• 14:09
  Speaker 2: And then thank you for sharing some of the links.
• 14:11
  Speaker 2: I'll I guess we'll handle the Q&A.
• 14:14
  Speaker 2: I'll keep going first.
• 14:16
  Speaker 2: But I think people always ask like yeah, so like, how do we use MPU?
• 14:21
  Speaker 2: How do we do that?
• 14:22
  Speaker 2: A lot of time I think you don't have to learn it.
• 14:24
  Speaker 2: That's a beautiful thing about what we're doing.
• 14:27
  Speaker 2: We try to simplify some of the developer workflow.
• 14:31
  Speaker 2: So then we handle the compression and handle the deployment for you and we put an open source library.
• 14:38
  Speaker 2: So if you really care about performance like you look at the clock, I have like only, let's say, two seconds to run this forecast you can actually go into the open source code and figure out where support and XR is needed.
• 14:51
  Speaker 2: In this case, with the library we use is called NNCF.
• 14:56
  Speaker 2: This is the neural network compression framework.
• 15:00
  Speaker 2: It does amazing work on making the model, for example, doing quantizations.
• 15:04
  Speaker 2: So from further two bits to in eight and four, let's say, a large language model.
• 15:10
  Speaker 2: We even do down to info with different algorithm behind it.
• 15:13
  Speaker 2: Why that's important?
• 15:14
  Speaker 2: I think there's a lot of techniques which I actually have a lot of tutorial.
• 15:19
  Speaker 2: Talk about those so we'll talk about this later in the slide so you can learn from what I talk about here.
• 15:23
  Speaker 2: Those so we'll talk about this later in the slide.
• 15:24
  Speaker 2: So you can learn from what I've been talking about here.
• 15:26
  Speaker 2: The key is with the optimization is basically means you're only use half of yourself, half your hardware most of the time or less, because under the hood, when we think about innate is not just the quantization, but under the hood the silicon has the computation unit specialized for the in a instructions.
• 15:46
  Speaker 2: So uh for sion, for example, on the cloud side, uh in a is a must for amx to activate it, which is the mx, the, the fianni at that time, and actually you can also use it for uh different point as well, but for great performance.
• 16:03
  Speaker 2: I think in a was the great starter because a lot of time I talked to real developer.
• 16:08
  Speaker 2: They didn't know you're not losing that much in terms of the accuracy.
• 16:12
  Speaker 2: Sometimes you may be doing better or some computer vision tasks.
• 16:15
  Speaker 2: It's weird, it just happens.
• 16:17
  Speaker 2: You know.
• 16:17
  Speaker 2: Sometimes it actually becomes a denoising to the model because doing the quantization something happened to it.
• 16:23
  Speaker 2: So you have to measure yourself to what you're doing.
• 16:26
  Speaker 2: But that actually is important for use cases where you may only get like 20 frames per second and opposite of that, 2 to 3X boosts you to real time and then you have a very different product as a outcome.
• 16:36
  Speaker 2: And on the edge use case, you cannot put three times more CPU or GPU on it.
• 16:42
  Speaker 2: Sometimes you may want to think about this, right, how do you get two times more?
• 16:46
  Speaker 2: It's actually a lot more expensive in terms of power, in terms of the cost.
• 16:49
  Speaker 2: So the software don't have an estimate.
• 16:52
  Speaker 2: And when we think about this, it's like when we do quantization, when we measure it, especially for YOLO those type, I think you just have to do it that's my word to it, depending on use case, of course, but that's definitely worth your time and then we profile the source code.
• 17:10
  Speaker 2: Here, for example, you can learn and we've done it in Magistrate for you why that's also important for large language model.
• 17:17
  Speaker 2: We talked about chatbot experience, right.
• 17:22
  Speaker 2: One problem that intrigued me a lot was when we started this a couple years ago I think almost two years ago when we first looked into the luxury language model.
• 17:32
  Speaker 2: The first problem we dealt with was not the computational problem.
• 17:36
  Speaker 2: First it didn't even run, it just blew up the memory, especially with GPU at that time, and then it stopped running at all.
• 17:45
  Speaker 2: So we started really figuring out the runtime, how to ensure that we're not making extra copies but at the same time we have.
• 17:52
  Speaker 2: The first step is if you look at the Lama 7 billion, right, if you look at the one 32 bits, I don't know if a mouse works on top of it it used 25 gig of RAM.
• 18:03
  Speaker 2: I don't even have any graphics card that closes that range, unless I go very high expensive ones, and they don't even work that well for 32 bits.
• 18:13
  Speaker 2: So there's a lot of gaps on what the world is giving you, especially on the model side, and what is deployable.
• 18:22
  Speaker 2: So at that time two years ago, it was an interesting time when we had to do a keynote.
• 18:26
  Speaker 2: So behind the scenes, I was one of the lucky ones to hack it, work with the driver team, working with the runtime team, working with the application team, to make one demo to run Lama.
• 18:35
  Speaker 2: And it worked.
• 18:38
  Speaker 2: When the first time I see the computer, the laptop, pumping out I don't know one token per second or something like that, I was like carrying my laptop, bring it to my wife and say oh my God, the computer is talking to me.
• 18:52
  Speaker 2: It's like a human, it's a bit slow, but like it's still answering my questions, be able to put a sentence together.
• 18:59
  Speaker 2: I think that's where I think there's a lot of like.
• 19:02
  Speaker 2: It's not just a hype, it's just like it's real.
• 19:05
  Speaker 2: It's actually solving some of the small problem.
• 19:08
  Speaker 2: But if you can solve these well and then we can collaborate, maybe you can solve it from.
• 19:14
  Speaker 2: I think that's how usually all these chatbots come together.
• 19:18
  Speaker 2: And that's exactly why we have info or even different algorithm.
• 19:22
  Speaker 2: This is actually doing asymmetric group 128, different ratio.
• 19:28
  Speaker 2: We want to measure the impact to the model when we compressing them and why that's important, I think.
• 19:34
  Speaker 2: At the end of the day, ram is ram.
• 19:36
  Speaker 2: You don't just double it overnight, but if you do write software, do the right algorithm, you can reduce it significantly and without huge performance impact into the model accuracy.
• 19:47
  Speaker 2: I think that's a win-win for what we've seen and that's why today, when you go to AIPC, a lot of model runs extremely well.
• 19:55
  Speaker 2: It depends on the engineering work.
• 19:57
  Speaker 2: If you, however, don't know if your model is working, we did put together a lot of different benchmarks.
• 20:06
  Speaker 2: That's what I love about Intel.
• 20:09
  Speaker 2: Sometimes we are very interesting about, like, how we approach things.
• 20:12
  Speaker 2: We just did the work and we didn't talk about a lot.
• 20:14
  Speaker 2: So in here we have one of the largest database of model.
• 20:19
  Speaker 2: Actually, if you go back into the, the IPC thread or some of the model, we have optimized thousands of them for sure.
• 20:25
  Speaker 2: But if you search for just AIPC or some of the model, we have optimized thousands of them for sure.
• 20:27
  Speaker 2: But if you search for just AIPC, you can see how well some of these models can run.
• 20:32
  Speaker 2: Before you fine-tune your stuff or anything, you can predict the outcome, how well this model will run on your machines.
• 20:41
  Speaker 2: I think that's very powerful, especially today.
• 20:46
  Speaker 2: We also work with Microsoft, as I said, right, and many different partners.
• 20:49
  Speaker 2: Some of the model.
• 20:51
  Speaker 2: When we optimize, we actually benchmark for you.
• 20:53
  Speaker 2: So when I say benchmark means we don't just like, hey, it's run this fast, but we benchmark against different parameters.
• 21:01
  Speaker 2: So think about this is almost like a hyper parameter style, right?
• 21:06
  Speaker 2: You just explore all the options in the pool and then look at the impact towards the final outcome.
• 21:11
  Speaker 2: So you go back to the NNCF library.
• 21:14
  Speaker 2: If you look at our documentations, oftentimes you'll see even the model we pre-compressed, pretty much like we try to see where is the sweet spot for you.
• 21:27
  Speaker 2: So where do you find more of this?
• 21:29
  Speaker 2: You can go to hugging phase um, look at openfino um, and then we have a shared repository with some of the pre-compressed model and that way then when you start using intel hardware, you know that you're getting the best software.
• 21:41
  Speaker 2: Take a screenshot of this.
• 21:44
  Speaker 2: This will save you a lot of headaches Well, at least my headaches and it will come through very well when you are prototyping, when you are doing any sort of work related to Intel or not even Intel, maybe even ARM.
• 22:01
  Speaker 2: So I put together some examples so I don't just talk, I create a lot of examples If you're interested in trying what I just talked about, I put together scripts.
• 22:14
  Speaker 2: You can double-click and it will run.
• 22:15
  Speaker 2: It will install, it will download and it will run those three steps in a bat file.
• 22:20
  Speaker 2: So this is the link to it.
• 22:22
  Speaker 2: Enjoy the code.
• 22:23
  Speaker 2: I did that two weeks ago, one week ago.
• 22:25
  Speaker 2: It should still work one year later.
• 22:28
  Speaker 2: I don't know, maybe things will change.
• 22:29
  Speaker 2: But uh, last week I tested and it works really well on your laptops.
• 22:34
  Speaker 2: Make sure you have 32 gig of ram, 32, okay.
• 22:38
  Speaker 2: And when it runs, um, it not just run on cpu, this one will run on mpu as well.
• 22:44
  Speaker 2: So, um, when people ask like, would this run on NPU?
• 22:47
  Speaker 2: Yes, it does.
• 22:48
  Speaker 2: When do I use NPU?
• 22:50
  Speaker 2: Depending on scenarios?
• 22:52
  Speaker 2: So in this case I asked the machine to write a song about AI with emoji.
• 22:56
  Speaker 2: I don't know why, but it worked.
• 22:59
  Speaker 2: It's kind of fun.
• 22:59
  Speaker 2: You can sing with it.
• 23:00
  Speaker 2: So now it's got your limit right, you can be very creative.
• 23:06
  Speaker 2: Build your own DJ, sing a song for you.
• 23:08
  Speaker 2: I think it's fun for the kids these days because they will open up their imaginations.
• 23:13
  Speaker 2: And then for my one-year-old, I got to teach them how to use Gen AI.
• 23:16
  Speaker 2: Now, check out to do something fun, just like have an interesting learning experience, but not fully relying on it, I'm just saying just to have exploration.
• 23:25
  Speaker 2: I think that's where curiosity right, and if you're interested in what we just we just saw here, um, I also feel that we also build on, just I as a team of us, um be a rack demo where you will see how we bundle this with, uh, our partners, like um, I think that partner only this is like all the community of members, right, um, it's like ever anyone that are using the tool, um, like from land chain llama index.
• 23:54
  Speaker 2: We integrate with them.
• 23:55
  Speaker 2: So then you see our presence on openfino in those large model tools and that's important because, um, you don't have to relearn.
• 24:04
  Speaker 2: You can what you have, but you just have to make sure you enable the right components and that's gonna give you the accelerations.
• 24:10
  Speaker 2: And as a homework, I'm closing in two minutes we also have example on the edge side where you can do a GenTech workflow.
• 24:22
  Speaker 2: So this is real.
• 24:24
  Speaker 2: Real like you can talk to a chatbot locally, that you can order some paint and then I don't know why you want to paint your house today or not, but it's a great experience.
• 24:35
  Speaker 2: You can try to see what a chatbot can do locally.
• 24:38
  Speaker 2: This one's a bit slower because we have a lot of extra deep thinking process, so it means we use a very big size chatbot.
• 24:46
  Speaker 2: So it means like we use a very big size chatbot but as a silicon move forward.
• 24:55
  Speaker 2: This is pretty fast already, but it will get even faster on the edge.
• 24:58
  Speaker 2: Make sure you take a screenshot of this one and at the last minute I think I'll do some closing.
• 25:01
  Speaker 2: First, learn in notebooks.
• 25:03
  Speaker 2: That's where you learn a hundred examples, and we have open edge platform that you can use to also learn how to deploy this on edge.
• 25:13
  Speaker 2: So if you Google open edge platform from Intel, you will see it in GitHub as well as a closing as a benefit.
• 25:22
  Speaker 2: Right, I have six benefit here.
• 25:24
  Speaker 2: Two, I think you have to remember Memory.
• 25:27
  Speaker 2: If you don't have memory for PIN that's sufficiently well managed, it will not run well.
• 25:33
  Speaker 2: That's what we learned for the years of trying to optimize it.
• 25:36
  Speaker 2: We got a lot of speed up by learning how to manage our memory.
• 25:39
  Speaker 2: A lot of things are memory bound.
• 25:41
  Speaker 2: And learn how to deploy on different platforms.
• 25:43
  Speaker 2: You never know what people need.
• 25:44
  Speaker 2: Sometimes CPU is busy, sometimes the GPU could be busy.
• 25:47
  Speaker 2: Maybe use the NPU as you're uploading.
• 25:50
  Speaker 2: So we saw gaming as one example.
• 25:51
  Speaker 2: People want to use the NPU for analytics.
• 25:54
  Speaker 2: You don't want to touch your CPU or GPU and then having this as scalable.
• 25:59
  Speaker 2: I think use OpenEdge platform, as I said, is very important.
• 26:03
  Speaker 2: This is the key.
• 26:05
  Speaker 2: Please download and try to give us feedback.
• 26:07
  Speaker 2: We open source Getty, which is an amazing tool for training and deploying image model.
• 26:15
  Speaker 2: We open source many great tool like office sharing different, like Ash notes and all that All the things that I think will take you thousand, maybe hundreds of thousands of hours to build.
• 26:27
  Speaker 2: We put it together into a platform for you and if you try it, you don't like it, that's okay.
• 26:32
  Speaker 2: We'll be here listening and then we'll make it better.
• 26:34
  Speaker 2: That's what happened at OpenFino.
• 26:36
  Speaker 2: Right In the beginning it was a bit rough, but now it's very smooth edge.
• 26:40
  Speaker 2: That's all I can say.
• 26:41
  Speaker 2: If you're really interested in what we're doing in the last minute, contact with us.
• 26:46
  Speaker 2: I think I spend my 25 minutes if I look at my clock correctly.
• 26:50
  Speaker 2: Am I right?
• 26:54
  Speaker 1: Yeah, thanks, great talk, raymond, and thanks also to Ashutosh who followed with the different questions.
• 27:02
  Speaker 1: You know great talk, really.
• 27:06
  Speaker 1: Maybe I have a couple of questions for you, raymond.
• 27:11
  Speaker 1: Really you provided a lot of assets and a comprehensive description of OpenVINO, but you know there are different types of industries small, medium and large industries which are spending time and efforts on familiarizing with what you call conventional AI.
• 27:38
  Speaker 1: And so what's your take on about the messages for these guys, one instead of another to move forward on generative AI, why and how these guys should find time to use OpenVINO and the great methods you described.
• 28:01
  Speaker 2: So I think that's a very good question.
• 28:03
  Speaker 2: Like, it feels like we dropped the ball on the deep learning stuff that we did four years ago, five years ago, right, all of a sudden we have this Gen AI, everyone just talks about it, and what happened to the people who already spent a lot of money and time on the deep learning side?
• 28:17
  Speaker 2: I think, first is, if you're already like doing work that is functional, like license plate detection, we deploy so many of them.
• 28:25
  Speaker 2: We forgot, right, there's a lot of the smart city idea that we use for retail.
• 28:30
  Speaker 2: Uh, they work extremely well.
• 28:32
  Speaker 2: I think those, uh, I think we just need to make them better.
• 28:35
  Speaker 2: Um, so when I say vict means oftentimes, I see JNI keep be the one that opened up the flexibility of some of these approaches A famous one, I would say, for a lot of these.
• 28:47
  Speaker 2: Again, I'll go back to the YOLO one, because a lot of people know it you need to find your model before it works.
• 28:51
  Speaker 2: You need to do a lot of extra work to ensure the model is tuned to the application.
• 28:56
  Speaker 2: They're not as flexible.
• 28:57
  Speaker 2: Right Flexibility is not built into the architecture itself.
• 29:01
  Speaker 2: So for those use cases, I think people should start realizing that this new wave of these approach can create flexibility.
• 29:10
  Speaker 2: I think this is the step.
• 29:12
  Speaker 2: One is when I think of generative.
• 29:14
  Speaker 2: A lot of things are a lot more tunable.
• 29:18
  Speaker 2: I say tunable, like YOLO, for example.
• 29:21
  Speaker 2: If you do it right, you can automate the fine tuning step with the agent approach.
• 29:26
  Speaker 2: And then on top of that I saw the one called YOLO-E.
• 29:29
  Speaker 2: They actually combine the YOLO, which is like hey, you can recognize the duck and the cat.
• 29:34
  Speaker 2: What if you've never seen a banana before?
• 29:36
  Speaker 2: Now, what Right, it's like a banana, it's our track, so we can actually do a prom injection which not injection, prom base one.
• 29:43
  Speaker 2: They basically combine in the tool called Clip, so then they can now recognize something that looks like a banana and then you can say banana on a table.
• 29:51
  Speaker 2: It's like it almost looks like a generative AI.
• 29:54
  Speaker 2: It's like you're generating the model, kind of like structure on the fly, but using the YOLO architecture.
• 30:00
  Speaker 2: So it's very effective, very efficient.
• 30:02
  Speaker 2: So I see the industry should very think about how can I create better efficiency on some of the older stuff we have done?
• 30:08
  Speaker 2: I don't think they obsolete, I think they are still serving a purpose.
• 30:12
  Speaker 2: But how can we reduce some of the time spending on that?
• 30:16
  Speaker 2: And then also I think my title was chatbot how can make it cumulative, make things more interactive?
• 30:23
  Speaker 2: The whole point is not like I don't care if it's a genetic or not.
• 30:26
  Speaker 2: Can I make a good experience from the user?
• 30:29
  Speaker 2: It's a number one rule, and before, with the older one, they're very limited.
• 30:33
  Speaker 2: Now we can create experience.
• 30:35
  Speaker 2: This is, I think, the key turning point.
• 30:37
  Speaker 2: I never felt so alive when I saw the machine talking to me.
• 30:40
  Speaker 2: Right, I think my kid will be like yes, dad, I saw the machine always talking to me.
• 30:44
  Speaker 2: It's like, but I spent the whole 40 years of my life never seen machine talking to me so I don't know people forgot, like right, so it becomes a new norm, but like back then the machine doesn't talk, they just, you know, jiggling on things.
• 30:57
  Speaker 2: I don't know what they're talking about.
• 30:58
  Speaker 2: Anyone use Microsoft Word as a little icon with the cookie things Just never answered my question when I started computing.
• 31:03
  Speaker 2: Anyone use Microsoft Word?
• 31:05
  Speaker 2: At one point there's a little bit help icon.
• 31:07
  Speaker 2: Anybody help me?
• 31:08
  Speaker 2: Not once.
• 31:08
  Speaker 2: But I'm not saying Microsoft was bad, but that time I thought AI was broken because I asked well, how do I do this?
• 31:15
  Speaker 2: It will give me a wrong answer all the time, but now we're getting to close it with the right answer.
• 31:19
  Speaker 2: Close and close every day, anyway.
• 31:21
  Speaker 2: So that's my take.
• 31:24
  Speaker 1: That's fantastic.
• 31:25
  Speaker 1: Thanks, Raymond, for elaborating so much.
• 31:28
  Speaker 1: Maybe another question more related to OpenVINO question more related to open vino imagine I am a young professional and I would like to leave my footprint designing a generative ai model.
• 31:48
  Speaker 1: How open vino allows me to ease this task?
• 31:53
  Speaker 2: to start.
• 31:53
  Speaker 2: I think that the starting point with openVINO is really again like.
• 31:59
  Speaker 2: Openvino is like a step forward to make sure that you get the best performance out, but I think over the year we change a little bit right.
• 32:10
  Speaker 2: We actually have the most number of examples you ever need.
• 32:12
  Speaker 2: If you look at the notebooks, we have about almost 200 of them, and we have our reference design kit.
• 32:17
  Speaker 2: We have another 20 of different examples.
• 32:19
  Speaker 2: I think we built one of the most comprehensive library of things.
• 32:25
  Speaker 2: Like anything you talk about you try it.
• 32:28
  Speaker 2: Most likely you've seen one copy of that on OpenVINO.
• 32:32
  Speaker 2: I think that's like.
• 32:34
  Speaker 2: It's almost like.
• 32:35
  Speaker 2: It gives you that library of material where you can dream without like.
• 32:41
  Speaker 2: When I was doing my PhD, I spent two weeks just to figure out how to run the CUDA code, just to get it running, not even just getting it performing.
• 32:48
  Speaker 2: It's just getting it running, and then I spent two months to get it performing and then I spent a year to make it actually publishable right, so there's a line for idea to executable to actual deployable.
• 33:00
  Speaker 2: It takes a year at that time when I started.
• 33:02
  Speaker 2: And of course, the Kuda is step zero, right, but similarly here, right, but with OpenFino, I think in a couple of weeks I think you'll be able to pull out some amazing demo.
• 33:13
  Speaker 2: Like I've got to tell an industry secret here.
• 33:17
  Speaker 2: Every demo I build takes only two weeks because I have to catch up with the industry.
• 33:22
  Speaker 2: Every single thing is two weeks.
• 33:24
  Speaker 2: In two weeks I can spin something out.
• 33:25
  Speaker 2: How Do I write from scratch, optimizing everything from scratch?
• 33:30
  Speaker 2: No, I have to use a tool.
• 33:31
  Speaker 2: This is a secret.
• 33:32
  Speaker 2: If you are starting new today, start with a fresh ground where you can be competitive with others.
• 33:37
  Speaker 2: I think that's a great starting point and that's why we're doing this.
• 33:41
  Speaker 2: Go to OpenPython Notebooks, make a contributor.
• 33:45
  Speaker 2: So we started with five contributors, including myself.
• 33:48
  Speaker 2: I have 120, something like that.
• 33:50
  Speaker 2: If you want to have a contributor, do Google some of the code with us, like we had some student programs.
• 33:55
  Speaker 2: So learn with the expert.
• 33:56
  Speaker 2: There's a lot of things you can engage with us.
• 33:58
  Speaker 2: I think at Meet your LinkedIn, you will see it.
• 34:01
  Speaker 2: There's a lot of different posts I made about how to engage with Mintel.
• 34:08
  Speaker 1: Yeah, that's great, Also the fact that you mentioned the students' programs, because we need educational contributions for people who are approaching.
• 34:20
  Speaker 2: Yeah, we've done it for four years straight, so, and this is gonna be fifth year next year, so you no-transcript.