Latest / EDGE AI POD / Cloud to the Edge: Future of LLMs w/ Mahesh Yadav of Google

Transcript

Cloud to the Edge: Future of LLMs w/ Mahesh Yadav of Google · EDGE AI POD

• 0:37
  Pete: Thank you All right, ta-da Fancy intro there, cool Yay.
• 1:03
  Pete: Hey, jenny, how's it going?
• 1:07
  Jenny: Good how are you?
• 1:07
  Pete: Thanks for having me again.
• 1:08
  Pete: Good evening, jenny from edge impulse, based in amsterdam.
• 1:11
  Pete: So we're this is a worldwide broadcast belview, washington, and, um, yeah, cool.
• 1:18
  Pete: Thanks for for joining us.
• 1:20
  Pete: We have a really interesting guest today, mahesh ad from Google, although we'll hear about his background because he also worked at Facebook and AWS and Microsoft.
• 1:29
  Pete: But before we get into that, I had a couple of housekeeping things to announce, by the way.
• 1:36
  Pete: So at the end of this little intro we have our sponsors, the people that support the TinyML Foundation.
• 1:41
  Pete: We have a new one that the logo didn't make it onto, the little thingy-dingy called WeTech.
• 1:45
  Pete: So I just wanted to give a shout out to the folks at WeTech.
• 1:48
  Pete: They're based in Scotland.
• 1:49
  Pete: Thank you, wetech.
• 1:51
  Pete: And they do all kinds of cool stuff around kind of industrial edge AI from kind of semiconductors and devices and systems to do that.
• 2:04
  Pete: So check them out.
• 2:04
  Pete: Wetech, w, e, e, t E, q.
• 2:06
  Pete: And the other thing is, uh, if you're not, uh, making a plan after this show to head on down to the computer museum in mountain view to check out edge impulses event today, you should do that.
• 2:19
  Pete: Um, they have their annual, um, what's it called Edge Impulse?
• 2:24
  Jenny: Edge Impulse Imagine.
• 2:25
  Pete: Imagine yes, that's happening today and it's free.
• 2:29
  Pete: So if you're in the Bay Area, you should go down there.
• 2:32
  Pete: It's a lot of fun.
• 2:32
  Jenny: It's in person and online.
• 2:34
  Jenny: So after this, just try and imagine.
• 2:38
  Pete: Just switch on over.
• 2:39
  Pete: There you go.
• 2:40
  Pete: Speaking of things to do in Austin, texas, in October 8th through the 10th, we have Embedded World North America.
• 2:49
  Pete: I'll be there.
• 2:49
  Pete: A bunch of folks will be there from the community.
• 2:51
  Pete: We have a couple of days where the talks about state-of-the-art Edge AI Tony ML you'll hear from everybody there about that.
• 2:59
  Pete: I'm actually doing a fireside chat with Nakul Dugal from Qualcomm, who runs their IoT and automotive and cloud group there, so that'll be a lot of fun and we're hosting the IoT Stars party on October 8th.
• 3:14
  Pete: That's always a really fun party, yeah, yeah that'll be fun, so you know lots of food and drinks and stuff, so meet us there.
• 3:21
  Pete: And then we also have online speaking of online, october 17th Gen AI at the Edge Beyond LLMs and Chatbots, the Journey to Generative AI at the Edge, free October 17th, on all the kind of a day long set of talks.
• 3:36
  Pete: In fact, we may even make it two days.
• 3:38
  Pete: We had so many amazing submissions of papers from around the world on people kind of breaking such new ground.
• 3:45
  Pete: We're going to talk about language model evolution today, but this is all about kind of LLMs and VLMs and Gen AI and the edge and what's happening there.
• 3:53
  Pete: So check that out.
• 3:54
  Pete: October 17th.
• 3:55
  Pete: You can register.
• 3:56
  Pete: I'll put the link in the chat, but it's also on our webpage.
• 4:00
  Pete: And then finally, if you're really planning ahead, november 6th we have an event in Taiwan Taipei 2025.
• 4:10
  Pete: And we have a ton of great speakers all celebrating all of the innovation happening in the Taipei Taiwan community around AJI on November 6th.
• 4:19
  Pete: So check that out too.
• 4:22
  Pete: Anyway, okay, you know we're done with the summer, so now that's a now everything is like full blast, right.
• 4:27
  Pete: Everyone's like let's just do stuff.
• 4:30
  Pete: So, exactly, summer's over, back to work.
• 4:35
  Pete: Speaking of work, let's bring on Mahesh and talk about all the cool stuff that he's been working on.
• 4:43
  Pete: Let's see here.
• 4:43
  Pete: Ah, there he is Ta-da.
• 4:46
  Pete: So welcome, mahesh, to TinyML Talks.
• 4:50
  Pete: Thank you, pete, thanks, danny.
• 4:53
  Pete: Awesome, so full disclosure.
• 4:56
  Pete: So Mahesh and I used to work together at Microsoft, but a lot of people have worked with Mahesh because he has worked at Microsoft and AWS and Facebook and now Google Right.
• 5:05
  Pete: So are you at the Googleplex right now or where are you calling in from?
• 5:10
  Mahesh: Yeah, I'm calling from Google Sunnyvale office.
• 5:13
  Pete: All right, sounds exciting.
• 5:14
  Pete: So, what are we?
• 5:17
  Mahesh: talking about Close to the computer museum.
• 5:19
  Pete: Oh yeah, that's right, You're real close.
• 5:22
  Mahesh: Oh, very close.
• 5:24
  Mahesh: So today we're going to talk about you, talk about where me and Pete go a long way.
• 5:28
  Mahesh: I think Pete used to be almost a VP then and then I was just a junior developer and Pete used to just come.
• 5:38
  Mahesh: Pete, somebody who reported to me, Lee used to report to Pete and he used to coach me a lot, and then we used to talk about Pete.
• 5:45
  Mahesh: So when Pete reached out, I was like, okay, what can I do for Pete, thank you.
• 5:52
  Mahesh: So I worked with Pete bringing small devices to life at Microsoft, but then I switched to large language model.
• 5:59
  Mahesh: I think a lot of people did switch that.
• 6:01
  Mahesh: So when Pete reached out, I thought it would be great if I can revise what we used to do with large language models, talk about small language models, why they are needed, and then bring some of the small language models to life using media pipe, tiny ML.
• 6:17
  Mahesh: And one thing disclaimer, disclaimer, it's not really tiny embedded AI, but good enough, right?
• 6:24
  Mahesh: It means 405 billion parameter model that can run in a 2 billion footprint onto your machine without internet.
• 6:32
  Mahesh: That's what we'll try to achieve today, and maybe next session we can do even further to reduce this to 20 MB and run it on your watch.
• 6:40
  Mahesh: But today we're going to just focus on 2 billion and see how can you actually run accelerated with local CPU kind of hardware at speed on your device using TinyML.
• 6:53
  Pete: Cool.
• 6:53
  Pete: Yeah, well, tiny is a relative term.
• 6:55
  Pete: We always like to tell people that.
• 6:56
  Jenny: So it's like Pete and I were just talking about before you joined, about how TinyML is an evolving term and in the future it'll mean so many different things or it'll be changing, and so yeah, so today is more so at the edge, rather than tiny, tiny yeah, exactly, that's still cancer flow, right?
• 7:14
  Pete: not cancer exactly so that's cool, that's cool, so um anyway.
• 7:20
  Pete: So why don't we bring up?
• 7:21
  Pete: Do you want to bring up your screen here?
• 7:23
  Pete: I can do that and yes, anyway.
• 7:25
  Mahesh: So why don't we bring up?
• 7:25
  Mahesh: Do you want to bring up your screen?
• 7:26
  Pete: here.
• 7:26
  Pete: I can do that and yes, and I can get started.
• 7:27
  Pete: You may fade to the background.
• 7:28
  Pete: I will encourage people who are online.
• 7:29
  Pete: I know I see a bunch of people.
• 7:31
  Pete: Uh, you can send in comments and questions and we will not try not to interrupt you too much, but occasionally we may break for for a few.
• 7:39
  Mahesh: Yeah, let's take a breaks in between and we can take questions sure sounds, Sounds good.
• 7:43
  Pete: Let me see if this is your screen sir.
• 7:46
  Mahesh: Yes, this is my days with Pete.
• 7:48
  Mahesh: Pete and me worked together on that small white device.
• 7:51
  Mahesh: We built this small Vision AI Dev Kit.
• 7:54
  Mahesh: The idea was that you can actually bring AI to Edge from Azure.
• 8:01
  Mahesh: We had a lot of Cloud, but we did not have an edge offering, so we built the first developer kit and this developer kit was running Qualcomm Snapdragon processor and NPU and the challenges this is 2018, the number of challenges bringing this to life and I was the lead or TL on this and the challenge we faced was that there was architectures like YOLO which Qualcomm did not support.
• 8:26
  Mahesh: It Everybody starts with NVIDIA, lives with NVIDIA operators, converts them to CUDA, runs on GPU, but if you want to run in on anything else, it doesn't work.
• 8:34
  Mahesh: And to solve those problems, what we turned into is something in-house which is customvisionai.
• 8:41
  Mahesh: So if you don't know what custom vision AI is, this is service Microsoft's built and what it can do for you is that you can.
• 8:48
  Mahesh: Actually it's pretty straightforward.
• 8:51
  Mahesh: You can just come in, give it few images and create a new project and just name the project and then what you can do is you can select object detection.
• 9:02
  Mahesh: But if you can select this compact model, you can actually select that Vision AI DevKit and in a click of a button you can just train a model and deploy on that hardware.
• 9:13
  Mahesh: So that is, we are restricting, combining the user problem.
• 9:18
  Mahesh: The user problem is that, hey, we have the images, but if we train with some open source architecture like YOLO, then we can't run it on your device.
• 9:26
  Mahesh: Let's take that problem away.
• 9:28
  Mahesh: The user have images and they're taking their images and creating models, and then we can figure out what architecture to use and then do the conversion, working closely with Qualcomm.
• 9:39
  Mahesh: So that's the one solution.
• 9:41
  Mahesh: This is, by the way, just the beginning.
• 9:43
  Mahesh: We're not even started.
• 9:44
  Mahesh: I'm just setting the grounds.
• 9:46
  Mahesh: That was one problem.
• 9:49
  Mahesh: This is the preamble, this is the preamble, it's all in that way.
• 9:53
  Mahesh: But then I realized to take it to production, to Azure scale, we needed to do more, and that's where we build Vision on Edge.
• 10:00
  Mahesh: This is a project which I worked on where, after doing the normal talks on Vision, ai, duckkit and hardware, this is where actually you can take any model, any camera which can stream RTSP, which is any $30 camera from these days, and you can just actually take the stream, train the model using custom vision and actually deploy it.
• 10:23
  Mahesh: All this is open source.
• 10:25
  Mahesh: And then we had these scenarios like counting object, employee safety, defect detection in your product on a factory floor, and all this was available and is available for free.
• 10:36
  Mahesh: You can read this blog.
• 10:37
  Mahesh: The link is in the slide deck so you can try it.
• 10:40
  Mahesh: This was made possible at this time.
• 10:43
  Mahesh: We also invested in ONIX.
• 10:44
  Mahesh: Onix is this intermediary layer.
• 10:46
  Mahesh: So the problem was that to run these models on the edge, you need to run acceleration.
• 10:54
  Mahesh: Acceleration means you need a hardware which can do matrix multiplication faster.
• 10:59
  Mahesh: So GPU is that one hardware, but there are others.
• 11:03
  Mahesh: Right, there are NPUs, there are other, like CPU with acceleration, with SNPE from Intel, but you need to take care of converting and running it on the edge or accelerating.
• 11:16
  Mahesh: Onix took care of that.
• 11:17
  Mahesh: Onix said train the model in TensorFlow, pytorch or any other platform and give the model to me.
• 11:24
  Mahesh: I will convert it into onix file and then we work behind the scenes with Qualcomm, intel and others to make sure if it is ONIX format, then automatically it runs accelerated on their chips.
• 11:37
  Mahesh: So then you can replace the chips from CPU to their GPU, to NPU, vpu, any other processor on Edge, and it run accelerated.
• 11:47
  Mahesh: So as a developer, you need not to worry about it.
• 11:49
  Mahesh: So that was the cool thing.
• 11:51
  Mahesh: I think that's the last cool thing I did on Edge and then I switched.
• 11:56
  Mahesh: I changed my life to an LLM.
• 11:59
  Mahesh: By the way, all this here is a lab.
• 12:01
  Mahesh: I just left it for you today.
• 12:02
  Mahesh: So here is a lab you can use.
• 12:04
  Mahesh: This lab covers everything I just talked about.
• 12:07
  Mahesh: It helps you what custom vision is.
• 12:10
  Mahesh: It tells you what transfer learning is.
• 12:12
  Mahesh: Here is a demo of building your own network and then, when you are done, here is a GitHub repo which you can use to deploy any model with an RTSP stream on your Edge device with some production scale solutions.
• 12:26
  Mahesh: So just follow it.
• 12:27
  Mahesh: If you run into problems, let me know.
• 12:29
  Mahesh: But just take any $30 camera and now you can actually build a project with AI on Edge.
• 12:36
  Pete: By the way.
• 12:36
  Pete: So people are asking to share links at some point.
• 12:39
  Pete: If you can throw them in the chat at some point, people will do it afterwards.
• 12:44
  Mahesh: Sounds like a plan.
• 12:45
  Mahesh: We'll just put all the links.
• 12:46
  Mahesh: I think yeah.
• 12:48
  Mahesh: So they are part of the PDF.
• 12:51
  Mahesh: It should be there, but if not, then all these are like links are here embedded, but if not, we'll paste it.
• 12:59
  Mahesh: Cool, sounds good.
• 13:02
  Mahesh: So what is the agenda for today?
• 13:03
  Mahesh: So that was just a preamble.
• 13:05
  Mahesh: So I thought it would be cool if we can all understand why LLM is such a driving force in the world, what is the core value they are offering and what is the difference that they bring, why they are good for edge or small language models.
• 13:19
  Mahesh: We'll see the need and we'll see what actually the innovation in large language model can contribute to edge or is already contributing to edge.
• 13:27
  Mahesh: Then we will take a push towards SLMs and understand the landscape.
• 13:32
  Mahesh: Like I will just show you a few leaderboards and show you all the excitement, because it's not that like we only talk about it.
• 13:43
  Mahesh: It's basically a very strategical pivot for a lot of companies Like look at, like microsoft.
• 13:46
  Mahesh: Microsoft is saying, okay, we're going to fund open ai, but we are not building a large language model, we're going to build a small language model called phi, and then they are investing in that.
• 13:56
  Mahesh: And same is if you look at google, google is investing in gamma.
• 13:59
  Mahesh: By the way, everything I'm saying is my opinions are my own, so I can talk freely across these companies and not get into this competitive challenge.
• 14:07
  Mahesh: So these are all my opinions, not like Google or Microsoft.
• 14:10
  Mahesh: And then we will take a small language model, we'll take Google Gamma model, we will convert the model and we will run it on edge without internet at high speed.
• 14:23
  Mahesh: We'll do a lab and then I can take some Q&A.
• 14:26
  Mahesh: That's the plan.
• 14:27
  Mahesh: I need to leave at 8.55 because I need to run to other building, but hopefully we have enough time to cover everything.
• 14:34
  Mahesh: With that note, I think we can get started.
• 14:38
  Mahesh: So the beautiful part about like these models have been there but the emergent behaviors, which is what happens in large language models, is that if you keep adding more and more parameters, they continue to show new capabilities on their own.
• 14:57
  Mahesh: So you keep throwing unsupervised data towards them and then you can see that they only don't understand language.
• 15:03
  Mahesh: They start doing summarization at more parameters are added or the size of model is increased.
• 15:09
  Mahesh: They can start doing logical reasoning, they can start doing qna, they can start doing tasks like proverb or translations out of box, so that that is the thing that people took on and the world gravitated towards LLM is that, hey, can I do more compute at it?
• 15:28
  Mahesh: And if I can do more compute and if I can figure out how can I add more layers and crawl all the Internet data, then actually I don't know what can I build.
• 15:38
  Mahesh: And this is what happened from GPT-3 to GPT-4 to next GPT-5.
• 15:42
  Mahesh: This is what happened to llamas of the world, from 7 billion to 27 billion to now 405 billion.
• 15:49
  Mahesh: And each time you add more parameters, you're seeing these models are becoming more and more capital.
• 15:56
  Mahesh: So that's the race, that's an arm race.
• 15:58
  Mahesh: Right now we don't even know, nobody knows, if we continue to increase the size of these models, what more can they bring to life.
• 16:05
  Mahesh: So right now we are at in open source, we are at 405 billion parameters.
• 16:10
  Mahesh: Nobody else is exposing how many parameters they are training on, but you look at 405 billion parameters.
• 16:19
  Mahesh: Just to give you an idea, what is a parameter?
• 16:21
  Mahesh: If you take an equation which is a simple equation of, of any llm is y is equal to wx plus b, which is y is your input.
• 16:32
  Mahesh: What you input, which is your prompt.
• 16:34
  Mahesh: Let's say, explain tiny ml and x are the.
• 16:38
  Mahesh: So y is the output, x is your input, which which you input, and these are the weights and biases.
• 16:44
  Mahesh: This is the layers of the model.
• 16:47
  Mahesh: So if you have a neural network, you can think of these are the layers and every time you connect.
• 16:53
  Mahesh: So this is your input and this is your output, and these are the weights.
• 16:59
  Mahesh: And every layer connects to another layer.
• 17:01
  Mahesh: So there are n number of weights going from one layer to another and the sum of these weights and vices in a model.
• 17:11
  Mahesh: This is a matrix, and this is a matrix is called parameters.
• 17:17
  Mahesh: When somebody says that, hey, I have this 5 billion parameter model, thank you whoever did that.
• 17:24
  Mahesh: But when we say that we have a 3 billion parameter model, thank you whoever did that.
• 17:28
  Mahesh: But when we say that we have a 3 billion parameter model, that means it has 3 billion weights and biases in this neural network, right?
• 17:36
  Mahesh: So that's what they're trying to say.
• 17:38
  Mahesh: If we have, and then if we more parameters, we can learn more capabilities.
• 17:43
  Mahesh: This parameter is actually the size of the model.
• 17:47
  Mahesh: There is a simple equation that we can learn, which is hey, if you have a three billion parameter model, what it means is it will have three billion numbers to store, right, if you have three billion numbers to store, then what you can do is, if you are using a 32-bit, like float operations, then you can just multiply that by three.
• 18:15
  Mahesh: That means you need a 12 GB RAM to store this model right.
• 18:23
  Mahesh: And then we will talk later in the call that how can you reduce this from four bytes float to an int or a full floating, 32-bit to 16-bit float and all, which is the optimization technique we use.
• 18:38
  Mahesh: So that's basically the groundwork of large language model.
• 18:41
  Mahesh: The idea is that the more parameters you continue to add, the more it continues to learn and the more capabilities it is showing.
• 18:50
  Mahesh: So that's what a large language model means, or that's the world of large language models, which is, you keep on adding these parameters, more weights and more vices, and you continue to do more and more with the same model.
• 19:04
  Mahesh: So if that's the land, then what are the challenges?
• 19:08
  Mahesh: Why would you even think about small language model?
• 19:11
  Mahesh: One is cost.
• 19:13
  Mahesh: You can think of Amy Hood who is counting money that they're giving to OpenAI right now.
• 19:19
  Mahesh: That's what it takes.
• 19:21
  Mahesh: I think last I read is 5 billion losses, which is the 5 billion losses for OpenAI.
• 19:30
  Mahesh: They have more than 100 million subscribers.
• 19:33
  Mahesh: They have free compute from Azure and still they are in 5 billion in losses just because these are really, really costly.
• 19:39
  Mahesh: 26 million it takes to run one cycle of 175 billion parameter model to train.
• 19:47
  Mahesh: Now, that was GPT-3.
• 19:49
  Mahesh: If you look at GPT-5, it has already gone to hundreds of millions to train one cycle of this model and they have to train many.
• 19:57
  Mahesh: And second is privacy.
• 19:58
  Mahesh: Enterprises still have a problem, like I talked to people in my previous roles, where they were trying to find insight in contracts, but cloud providers contract can only be read by three people in a company.
• 20:14
  Mahesh: Forget about, like uploading that contract on internet and then letting anybody read it.
• 20:19
  Mahesh: So how will you secure this data and a lot of things that you want AI to run on, but you're not.
• 20:26
  Mahesh: Like I need a headshot for my boys, but am I comfortable uploading their pictures on some website, even if I pay them, even if they say that they're not going to use them?
• 20:37
  Mahesh: I know they can be hacked easily, so are you going to leave that there?
• 20:41
  Mahesh: And then you hear news every day where you can just take images, and there was recently in EU there was a case where people took images and then it was so easy to just strip clothes out of them and circulate them on internet.
• 20:57
  Mahesh: So that's the kind of threat we have highly cost.
• 21:00
  Mahesh: And second is ensuring privacy.
• 21:03
  Mahesh: Once your data leaves your devices is 100x more difficult than securing it on your device.
• 21:11
  Mahesh: So these are the two drivers why we will have challenges, or why people even need small language models.
• 21:18
  Mahesh: And what is the advantage?
• 21:20
  Mahesh: Right?
• 21:21
  Mahesh: So if you're gonna go this, so we have the disadvantage, but what is the advantage for edge, why it is so cool time to work on the edge, ai or do these small models?
• 21:31
  Mahesh: There are three means that I could think of.
• 21:34
  Mahesh: One was that these more large model models actually are fueling the growth for small language models.
• 21:41
  Mahesh: Small language models were always all around, like BERT small, bert mini, bert large powered, all the AI services.
• 21:49
  Mahesh: These are the services which do speech-to-text translation, these are the services which do text-to-text conversion, or these are the services which we use to find sentiment analysis.
• 22:01
  Mahesh: But now these large models can be distilled or, using these large models, we can actually convert them into small language models.
• 22:13
  Mahesh: So, with a fraction of cost, you can actually build an SLM which is almost as capable as the large language model and with 1000 of the cost.
• 22:25
  Mahesh: So that's a huge opportunity if you're invested or building SLMs and that's the opportunity you are seeing where FI is getting released or GAMMA is getting released, because the investment is already made and how can you actually convert them and give enterprises something that they can actually take, build and iterate on and then keep it as their own IP is a huge opportunity, I believe, just because we have large language models around.
• 22:53
  Mahesh: Second is earlier the challenges like getting synthetic data or generating data for training, your models was so bad Like when we built this custom vision one we were just doing augmentation but we couldn't get any data.
• 23:06
  Mahesh: But now you can go to mid journey.
• 23:09
  Mahesh: You can go to any other website and create an image just like that, which is like almost real images and same for text.
• 23:17
  Mahesh: You can just give your scenarios and generate a lot of data between these slm to your use cases.
• 23:23
  Mahesh: And third is because these large models are running on really high compute and there is a huge demand for them.
• 23:31
  Mahesh: There's a lot of innovation happening on how can we inference or how can we optimize the inference operation, which actually, because the underlying architecture is same, called transformers.
• 23:43
  Mahesh: You can actually take the same optimizations and apply it on edge.
• 23:47
  Mahesh: So, first, you make the.
• 23:49
  Mahesh: With first two, you can actually build a small model and third, you can actually take all the learnings we have done with optimizing inference on large language models to directly small language models, because the underlying architecture is same.
• 24:05
  Mahesh: So these are the three.
• 24:07
  Mahesh: I think those combined together are creating a really huge opportunity for everybody to innovate, build and do SLMs.
• 24:17
  Mahesh: And that's what you are seeing Like if you look at leaderboard.
• 24:20
  Mahesh: I was just checking before the call, I looked at this Hugging Face dashboard and I was looking at the benchmarks with a very small size of.
• 24:32
  Mahesh: Let's just reduce the size.
• 24:35
  Mahesh: So this is Hugging Face leaderboard, this is an LLM leaderboard, but you can reduce the size to, let's say, 1 billion parameters and you can see models and models from companies like Microsoft, stability, google, all in open domain and you can just take them and start building on top of them.
• 24:57
  Mahesh: So that's the kind of excitement out there, because those three things that I earlier talked about, that's the kind of ecosystem we are dealing with and, if we go back, there are three main techniques here Quantization, pruning, distillation.
• 25:13
  Mahesh: I can go more details in this, but I want to stop and take some questions, because questions are always interesting.
• 25:20
  Mahesh: And then I have this and maybe then we can jump to lab.
• 25:26
  Pete: Yeah, we have some interesting questions here.
• 25:28
  Pete: Let's see, let me throw something up here.
• 25:30
  Pete: One question about well, I guess the most recent one here are the models.
• 25:36
  Pete: They're instruction tuned.
• 25:38
  Pete: You want to speak to?
• 25:39
  Mahesh: that yeah.
• 25:41
  Mahesh: So instruction tuning is this technique where basically what you're saying is that hey, this model out of box is trained on somebody else's data internet data.
• 25:51
  Mahesh: It doesn't know my company, it doesn't know my customer.
• 25:53
  Mahesh: In this contract example I was taking, let's say, you upload a contract and you want to know what are the risks in this contract.
• 26:00
  Mahesh: But you have your lawyers who have already taken some contracts and given risks.
• 26:06
  Mahesh: So how can you use that data to tweak the parameters in the model that we have?
• 26:12
  Mahesh: That's what you take a good, you take question but also take good answers from your experts and then you tween the model to do the job.
• 26:24
  Mahesh: So out of box, these are not instruction trained for your scenarios, but they are trained on doing Q&A.
• 26:31
  Mahesh: So you have done fine-tuning so that they can do Q&A operations.
• 26:36
  Mahesh: So it's not only just the weights, you actually have the interface.
• 26:39
  Mahesh: If you send them a question they will answer.
• 26:41
  Mahesh: They're not like GPT-3 dumb.
• 26:42
  Mahesh: So we have all the open source model that you are getting from Microsoft, google are basically instruction to do Q&A.
• 26:51
  Mahesh: But there are other techniques.
• 26:53
  Mahesh: We do other lab in our classes where we show you how can you actually take any scenario like the contract example.
• 27:01
  Mahesh: Take the ideal Q&A and instruction to your model with techniques like low-rank adopters, lora, and you can do PEFT, which is parameter efficient fine-tuning, and then we show one technique, lora.
• 27:15
  Jenny: Mahesh, what courses are you referring to?
• 27:17
  Jenny: Are you referring to your ones on Maven or?
• 27:21
  Mahesh: Yes, oh, thank you.
• 27:22
  Mahesh: No, no, I just think you should really advertise yourself here?
• 27:26
  Jenny: I'm curious.
• 27:28
  Mahesh: No, there is a link in the slides.
• 27:30
  Mahesh: I teach a course on Mivin where I help product managers.
• 27:33
  Mahesh: I also have a community on LinkedIn and we meet every Saturday and we do labs or sessions like this.
• 27:40
  Mahesh: We try to stay ahead of the curve, so please join the community.
• 27:44
  Mahesh: Join the Saturday free sessions.
• 27:46
  Mahesh: Or, if you really want to learn how we are building this product, do a project end-to-end, join the Saturday free sessions.
• 27:49
  Mahesh: Or, if you really want to learn how we are building this product, do a project end-to-end, join the course.
• 27:52
  Mahesh: It's a five-week course on Mevit.
• 27:54
  Mahesh: Thank you.
• 27:57
  Pete: There's another question here from Chunyum this student-teacher approach for making a good SLM.
• 28:05
  Pete: Do you have any comments on that?
• 28:08
  Mahesh: That's a great.
• 28:09
  Mahesh: I will just cover it in distillation.
• 28:10
  Mahesh: I will show you how it is done and how can you actually do it.
• 28:14
  Mahesh: I will show it in code.
• 28:17
  Pete: This is a good segue, sounds good.
• 28:20
  Pete: Why don't you carry on?
• 28:22
  Mahesh: Perfect.
• 28:22
  Mahesh: So why don't let's look at the main challenges?
• 28:26
  Mahesh: As you saw, the main challenge we are dealing with is so can I just get my board?
• 28:31
  Mahesh: You guys have the slides so you can look at these three techniques, but let me explain on board what this means.
• 28:36
  Mahesh: Perfect Thanks, pete.
• 28:37
  Mahesh: So three techniques on reducing the size.
• 28:40
  Mahesh: So the problem we are having is that parameters are increasing to 405 billion parameters.
• 28:45
  Mahesh: We need models with 1 billion or less parameters.
• 28:50
  Mahesh: Ideally, what we are getting is 2 billion, which is good enough, right.
• 28:55
  Mahesh: So that's where we want to go and to reach.
• 28:58
  Mahesh: There are three techniques.
• 28:59
  Mahesh: One is you can quantize the model.
• 29:01
  Mahesh: Second, you can prune the model.
• 29:04
  Mahesh: Third, you could distill knowledge from these large models into small models.
• 29:11
  Mahesh: So quantization is the simplest one.
• 29:13
  Mahesh: This is what we did on our dev kit that I showed.
• 29:18
  Mahesh: So here as we discussed what are the parameters?
• 29:22
  Mahesh: Four billion parameters.
• 29:23
  Mahesh: Each is basically a variable.
• 29:25
  Mahesh: So y is equal to wx plus b.
• 29:28
  Mahesh: W is a matrix of all the weights.
• 29:32
  Mahesh: And what is a weight?
• 29:34
  Mahesh: Weight is just a number, which ideally is a float 32 number, right?
• 29:41
  Mahesh: So it takes four bytes to represent this number, right, 32 bits.
• 29:46
  Mahesh: What quantization says is hey, do you really need this.
• 29:50
  Mahesh: Such a big number, 32 bit is a lot of numbers.
• 29:53
  Mahesh: So can you reduce the precision from float 32 to float 16, or let's do int 8, or let's do int 4, or last I read is int2 or one bit quantization.
• 30:11
  Mahesh: So basically you are saying each quantization so this will reduce the size of your model by half right.
• 30:20
  Mahesh: So 405 billion parameter reach here becomes 200 billion right.
• 30:26
  Mahesh: And then if you do int eight it reduces by four.
• 30:29
  Mahesh: So two more hundred billion.
• 30:31
  Mahesh: If you go here, 50 billion.
• 30:33
  Mahesh: And then, if you go here, you reduce by four further.
• 30:36
  Mahesh: So if I can actually represent this model into this 405 billion parameters, I can actually get divided by four, which is you're looking at 12 billion.
• 30:46
  Mahesh: So just by quantization you can go from 4 billion to 12 billion.
• 30:50
  Mahesh: That's the power of quantization, each step you do.
• 30:54
  Mahesh: The only drawback is you're playing with the precision, and precision means accuracy.
• 31:00
  Mahesh: So your accuracy will go down and there are techniques.
• 31:03
  Mahesh: How can you survive the accuracy?
• 31:05
  Pete: And that's what a lot of people are doing.
• 31:07
  Pete: Yeah, Mahesh, there was one question that came in about one bit quantization.
• 31:11
  Pete: Is there such a thing as one bit quantization?
• 31:15
  Mahesh: yes, so that's what.
• 31:16
  Mahesh: There is a research paper on it and we looked at it.
• 31:19
  Mahesh: I can't share a lot, but one bit quantization is where the future is, so you can like look at that and that's what we'll need when we go to the embedded devices.
• 31:30
  Mahesh: So if you're looking at just binary classifiers, which says 0 or 1, which is saying, hey, is this an image of a cat or dog, or is it sentiment, positive or negative, all these things I can store in one bit and how can I reduce that?
• 31:45
  Mahesh: So that's the idea.
• 31:47
  Mahesh: And 2-bit, 4 bit is in production right now, but a lot of work is going in labs where we are trying to see how can we take it to even further.
• 31:57
  Mahesh: One bit, yeah, okay, carry on.
• 32:01
  Mahesh: Second idea is pruning.
• 32:02
  Mahesh: So you have all the weights, right, you have a lot of weights, and what you can say is, hey, pete can say is that hey, list all my weights.
• 32:07
  Mahesh: And what you can say is, hey, pete, pete can say is that hey, list all my weights and reduce 50 of them, lower 50, set it to zero or eliminate it from my network.
• 32:18
  Mahesh: So in pruning you are basically taking a 405 billion parameters and saying, just let go 200 billion weights which have a threshold you can define, and if it is less than that threshold you just let go 200 billion weights which have a threshold you can define and if it is less than that threshold you just let go those weights.
• 32:33
  Mahesh: That means you have pruned your model.
• 32:36
  Mahesh: You have actually reduced the size of the model by reducing the weights on converting them into zero, and then you have a sparse matrix that you can actually play around or run it faster using sparse inference techniques, around or run it faster using sparse inference techniques.
• 32:54
  Mahesh: And third is the distillation, which is the most used and most common technique that has been deployed.
• 32:58
  Mahesh: Pretty much every llm that you have, a small llm that you have seen.
• 33:02
  Mahesh: And the idea is that while training a small model, first start with the small parameters, start with two billion parameters, but during training give these guys a soft landing target.
• 33:16
  Mahesh: That, hey, by the way, here is the answer to this question.
• 33:19
  Mahesh: So, hey, here is data, here is what the target weight is for you to achieve.
• 33:26
  Mahesh: So each weight you are training.
• 33:28
  Mahesh: In training, what we do is we try to find the best value of W and B which can minimize this X, the value of Y, from the target Y, right.
• 33:41
  Mahesh: So this is what the ideal answer is and this is what your answer is.
• 33:47
  Mahesh: We're trying to reduce the cost function or the difference between these two, right?
• 33:49
  Mahesh: So You're trying to reduce the cost function or the difference between these two, right?
• 33:51
  Mahesh: So if you're trying to reduce the difference between these two, you're looking to change the value of weight and V, and this is the process of training, where you change these values.
• 34:01
  Mahesh: So in this one, what you do is because you already know the right weights right, for you run few things you get the right weights and during training you give them a target, you can give them a range that hey, by the way, this weight ranges in between, so try to converge there.
• 34:19
  Mahesh: So give him a cheat sheet, kind of thing.
• 34:21
  Mahesh: So once you give the cheat sheet, the model actually can learn with smaller parameters, can learn faster.
• 34:29
  Mahesh: And if we can switch to the screen, I can show all these three.
• 34:36
  Mahesh: I wrote a paper on this.
• 34:37
  Mahesh: This is a lesson we have which you can use now.
• 34:41
  Mahesh: So here we talk about quantization, which is straightforward.
• 34:47
  Mahesh: Here is code, this is pruning, and here is pruning code you can look at.
• 34:53
  Mahesh: And this is the distillation idea, where you actually take the loss function and you pass the soft target here.
• 35:00
  Mahesh: So let's look at in the code.
• 35:02
  Mahesh: So I explain this.
• 35:03
  Mahesh: This is how you do pruning Basically you take your model, so I define the model, then I compile and then this is the prune function where basically you take your model, so I define the model, then I compile and then this is the prune function where basically you are taking the threshold and then you are just removing the weights below this threshold and now you are saying, hey, give me sparsity of 50%, and then you check your, then you do training.
• 35:26
  Mahesh: You need to retrain after pruning to make sure your accuracy can come back.
• 35:31
  Mahesh: Because if you reduce just half of the weights and say imagine it will work, it won't, and then you prune and then you retrain and eventually you will find a point where you are happy with the accuracy but your size is also something you can manage.
• 35:44
  Mahesh: And this is the idea of a teacher and the student model, which is the idea of distillation.
• 35:49
  Mahesh: So you first have a large language model you use in this example we take GPT-2 as a large model and then we say, ok, let's generate a soft target, what that target looks like, to answer a question and then when I'm training the student model, I will pass this soft target during the training.
• 36:08
  Mahesh: So if you look at this train function here I'm saying, hey, computer, student and teacher probabilities, and here is your cheat sheet, which I got because I have a large model.
• 36:22
  Mahesh: So if we didn't build these large models, we didn't have these soft targets and it's impossible to find the kind of accuracy we are looking for to do the jobs like translation or do the job like Q&A.
• 36:33
  Mahesh: If we didn't have soft targets, you can't build it.
• 36:35
  Mahesh: So that's the advantage I was talking about earlier that having large models allows us to have techniques like distillation come to life, because here you can generate the targets using a large language model which OpenAI has invested in or Google has invested in, create your soft targets and then pass these targets during the training process so you can learn faster, better.
• 36:59
  Mahesh: So that's the ideas about three techniques.
• 37:03
  Mahesh: I think you can read more and if you have questions, obviously Saturday Community Sessions, you can see me again.
• 37:09
  Mahesh: But I wanted to spend more time sometime today in getting us into a lab where I can show you quickly, as how can you take actually a gamma model from a hugging face, convert it into tensor fluoride and then run it on your machine in like 10 minutes.
• 37:26
  Mahesh: But I think Pete is here.
• 37:28
  Mahesh: That means more questions.
• 37:29
  Pete: Well, we had a couple of questions here so we can take a little breather.
• 37:32
  Pete: Let me see, let's bring us back here.
• 37:35
  Pete: We got a couple of really good questions coming in.
• 37:37
  Pete: Well, here's one that's more of a general purpose question.
• 37:40
  Pete: Small language models are good at specific tasks, right Training on a very specific sets of data.
• 37:46
  Pete: What are the applications that you're seeing out there that these models are really applicable for?
• 37:51
  Mahesh: First of all, vivek is a friend.
• 37:52
  Mahesh: He was part of our classes, so thank you.
• 37:54
  Mahesh: Vivek Seems like a CD, he's a plant, no, no, but Vivek is always very curious.
• 38:01
  Mahesh: So I think we never discussed actually a lot about SML.
• 38:04
  Mahesh: So I invited all of our people to join and say let's discuss small language models for a change.
• 38:11
  Mahesh: But I think what kind of application these models are applicable for?
• 38:14
  Mahesh: So I think one application which always works, which I have seen a very stickiness with enterprises especially, is that they want to take a large model but they want to store their own learnings.
• 38:25
  Mahesh: So when they do the modifications, like in the contract example again, so yes, I will use an SLM, I will use a large language model, but when it does wrong, my lawyer goes and corrects it.
• 38:40
  Mahesh: We want to store that and then we don't want to give it to some open AI to go and train that model.
• 38:45
  Mahesh: We want to use an SLM to train this and it won't be perfect, but eventually, after six months of doing it, it will be awesome.
• 38:55
  Mahesh: On my company learnings and then company specific things I will pass to my slm and then everything generic I will pass to large language model.
• 39:04
  Mahesh: The goal is in three to five years.
• 39:07
  Mahesh: My small language model, with the reduced cost where I own the ip, knows so much about my company that it beats the llms and I need not to pay this expensive price, because the price we are getting for large language models today don't expect it to stay.
• 39:24
  Mahesh: It's like uber you're right now getting a lot of vc money into your pockets channeled through openai, but eventually it will stop and when it stops then do you have a model or not?
• 39:36
  Mahesh: Right?
• 39:37
  Mahesh: So that's a killer use case across the platforms.
• 39:40
  Mahesh: Second, what I've seen them like, that's just like a generic use case.
• 39:46
  Mahesh: But slms are really good at doing one specific thing which can be.
• 39:52
  Mahesh: You can do translations with it, you can do sentiment analysis with it.
• 39:56
  Mahesh: So anything an AI service can do, slm can do.
• 40:00
  Mahesh: But if you're trying to do a lot of generic things with them that hey, one day become this, tomorrow become this, then it won't work.
• 40:07
  Mahesh: So you will need one SLM for your contract data compliance, just that piece.
• 40:14
  Mahesh: You will need one SLM for that and one SLM for your marketing specific things.
• 40:21
  Pete: So that's the kind of things you need, I think there was a comment about don't try to replace your lawyer with an LLM, but that I'll let the-.
• 40:28
  Jenny: No, I completely agree 's another.
• 40:31
  Pete: another question here from harsha about you know, obviously we're talking about lots of different uh processors that could be used and there's like no shortage of them.
• 40:40
  Pete: Um, but power and energy considerations I mean you mentioned one of the things that I know is that a lot of these, you know, running an llm on an edge platform is not always kind of tops bound per se, but it's like memory bound in many cases, right?
• 40:54
  Pete: So can you put enough RAM in the device to really run the LLM?
• 40:58
  Pete: And that actually will drive up your costs, probably more than the processor itself, but I don't know if you had any comments on that.
• 41:03
  Mahesh: No, I think that's exactly right.
• 41:05
  Mahesh: What the USP of NVIDIA is that every time you look at nvidia new release, they double the ram because the world is doubling the size and, as we discussed earlier, that comes by.
• 41:14
  Mahesh: If you have a 2 billion parameter model and if you do it in full floats, which is 4 bytes, you need an 8 gb of ram.
• 41:23
  Mahesh: Now this size is going 10x every year, right?
• 41:28
  Mahesh: So so if it go 10x 20, then our RAM sizes are not catching up.
• 41:33
  Mahesh: Ram sizes are just doubling while the model size is going twice.
• 41:37
  Mahesh: So that, I think, is the biggest challenge in our world.
• 41:41
  Mahesh: But these are memory bound things and when you're considering hardware, right, one beautiful thing that is happening across the ecosystem for all of us is that, uh, the there are libraries now which one we will talk about?
• 41:57
  Mahesh: Media pipe, today, these, and earlier I talked about onyx, so I actually seeded my talk in the beginning.
• 42:03
  Mahesh: So what first you need is you actually need conversions of these large models to something that anybody can run, and that's what MediaPipe Tiny TensorFlow Lite does.
• 42:14
  Mahesh: A TensorFlow Lite converted model can actually run accelerated on M2 chips from Apple, can run it on Snapdragon chips across Intel NPUs or VPUs and obviously, our Nvidia GPUs.
• 42:36
  Mahesh: So first, the consideration is just buy more RAM, as Pete mentioned.
• 42:40
  Mahesh: So if you choose between 8 GB or 16 GB, even if $300.
• 42:44
  Mahesh: So I think RAM and a hardware processing unit, but I think we are living in a world, at least on the edge, like phones, or even your watch is coming with a AI accelerator chip in built in it, and then software like TF light and media pipe are allowing you to do the conversion, and once you convert it into TF light format or ONIX format, these are just taken care of.
• 43:10
  Mahesh: So just buy more RAM and make sure you know one framework which does the conversion.
• 43:15
  Pete: All right, buy more RAM.
• 43:16
  Pete: One more quick question too, is around ONIX.
• 43:19
  Pete: Where does ONIX fit into this whole equation?
• 43:21
  Pete: You mentioned it earlier about kind of a good intermediate recordability, but does it apply here?
• 43:28
  Mahesh: Okay, two minutes on ONIX, and then we will go to the lab.
• 43:30
  Pete: Okay.
• 43:31
  Mahesh: We'll leave the lab.
• 43:32
  Mahesh: I think lab is not interesting, but questions are.
• 43:34
  Mahesh: So lab will be there.
• 43:36
  Mahesh: You can run it.
• 43:36
  Mahesh: I will walk you through.
• 43:37
  Mahesh: Okay, so let's do ONIX.
• 43:39
  Mahesh: So what happened is.
• 43:40
  Mahesh: So what happened in the world is that you created a model with TensorFlow, which is from Google, pytorch another framework to build models, which is from Facebook or Meta, and then you have a tf file as a model or you have pt file as an extension.
• 44:01
  Mahesh: So your model looked like this.
• 44:02
  Mahesh: Now, if you wanted to run this on a GPU, it run out of box.
• 44:11
  Mahesh: This thing can run out of box because the maker of PyTorch and TensorFlow ensured that these models are designed in a way that because they use the NVIDIA GPUs to test it and these natively support Nvidia gpus.
• 44:30
  Mahesh: So one line code change and you can run them accelerated on nvidia gpus.
• 44:34
  Mahesh: But, as you will see, in enterprise or edge ecosystem, all your devices will not have nvidia gpus.
• 44:44
  Mahesh: Some will have intel vpus, some will have Intel VPUs video processing unit, some will have Qualcomm accelerator, some will have Apple M2 chips.
• 45:00
  Mahesh: So now what?
• 45:01
  Mahesh: Now the developer has to convert these models to run accelerated here, here or here.
• 45:10
  Mahesh: Onyx said don't do this.
• 45:11
  Mahesh: Onyx said that.
• 45:11
  Mahesh: Hey, you know what you can train in any architecture.
• 45:14
  Mahesh: And Onyx stands for Open Neural Network Exchange.
• 45:19
  Mahesh: And what they said is we will provide you a converter which takes a tf file or pt file and convert it into onix file.
• 45:29
  Mahesh: And if it is a onix file then out of box it get accelerated here because our operators, the intermediately form we created, is already supported by these chips.
• 45:42
  Mahesh: So they work behind the scenes with chip providers and then one single format can kill all chips or all ages.
• 45:52
  Mahesh: You get the power of acceleration.
• 45:53
  Mahesh: So these are the two problems they solved.
• 45:55
  Mahesh: One, they created one platform in the whole org to store models in one format.
• 46:01
  Mahesh: And second, if the models were stored, then they can be deployed on edge devices with acceleration without the developer doing any coding after they convert it to ONNX.
• 46:13
  Mahesh: They need not to optimize for these.
• 46:15
  Mahesh: So that's the idea of ONNX and that's what ONNX provides.
• 46:18
  Mahesh: In between, this is an intermediate relay representation which allows you to accelerate on any chip if it is open ONNX compliant, which pretty much every chip is, and that actually helped a lot of startups, because startups the biggest challenge is that developers don't know my code chain.
• 46:39
  Mahesh: They don't want to learn it, but if somebody buys my device, nobody's going to accelerate.
• 46:44
  Mahesh: So even if that chip is there in your device, nobody's using it.
• 46:47
  Mahesh: But if it is ONIX, then it works out of box and developers need not to do anything.
• 46:52
  Mahesh: So I think that is the cool part and that's why ONNX remains this driving force when Microsoft got it right.
• 46:59
  Pete: Yeah, they did and then.
• 47:01
  Pete: But I would say also, you know, since we're talking about devices and low power, I mean, in order to hit that power profile, you really do need to make sure your code is running on that acceleration part of the chip right, which is always kind of, like you said, the kind of the hard part.
• 47:14
  Pete: Because especially if you're a newer player, you know you're probably that model wasn't originally designed for it.
• 47:20
  Pete: But the only way to get the power profile, to get the battery powered or the solar powered, whatever, is to really make sure you're taking advantage of the npu or whatever it's called, vpu on the chip itself, so pretty important.
• 47:32
  Mahesh: I remember we had an army against NVIDIA where we were just launching different things at NVIDIA.
• 47:38
  Mahesh: Onix was one of them.
• 47:39
  Mahesh: But you know, this piece of ML and AI has gone so fast that people are not ready to wait, and every year there's something new that comes up.
• 47:48
  Mahesh: Where people say time is the currency, I will give Nvidia billions and billions of dollars because I can't wait six months After six months.
• 47:57
  Mahesh: Yes, amd will come.
• 47:58
  Mahesh: Yes, intel will provide.
• 47:59
  Mahesh: Yes, onyx will come along, but six months is a huge time when I'm a trillion dollar company and I have $200 billion in my bank, exactly.
• 48:09
  Mahesh: So that's the kind of deal we are dealing and hopefully one day we will see a reduction in that behemoth called Nvidia.
• 48:18
  Pete: My bet is against, but let's see Well it's great that they have really helped, you know, make the market and so, you know, definitely props to all that they're doing.
• 48:28
  Pete: We wouldn't be here probably having this show if it wasn't for the NVIDIA innovation that's been going on.
• 48:32
  Pete: So we appreciate that.
• 48:34
  Mahesh: No, no, of course I think whole AI won't be existing.
• 48:37
  Mahesh: So completely agree.
• 48:38
  Mahesh: But you know, just this idea that everybody can play the game and actually have a fair play is what I'm rooting for.
• 48:46
  Pete: Democratizing it is really important, so that people will get an opportunity, that word democratizing.
• 48:51
  Pete: Democratizing AI.
• 48:52
  Pete: What does it mean?
• 48:54
  Pete: Everyone gets to build models with their data sets and run it on their premises with their stuff, without writing a single line of code.
• 49:03
  Mahesh: So, nvidia, actually does that.
• 49:04
  Mahesh: If you look at it, nvidia does democratize AI because, as you saw, you did TensorFlow or PyTorch and if you're using NVIDIA chip it works.
• 49:12
  Mahesh: That means you have democratized it to a student in high school to run acceleration.
• 49:18
  Mahesh: But if you want to run it on my chip, then you need a PhD Developer in your team.
• 49:26
  Mahesh: So it's not democratized that way, okay.
• 49:28
  Mahesh: Last thing I want to land is the team.
• 49:29
  Mahesh: So it's not democratized that way.
• 49:30
  Mahesh: Okay.
• 49:30
  Mahesh: Last thing I want to land is the lab.
• 49:32
  Mahesh: So what we worked on is this lab.
• 49:34
  Mahesh: We like to leave something with people to try, so we created this simple lab.
• 49:39
  Mahesh: What it does is basically it does two or three things.
• 49:43
  Mahesh: One cool thing it does is it allows you to download our Google Drama model latest 2 billion model and this is quantized at 4-bit in int4.
• 49:56
  Mahesh: So already you can see reduction of 16x and then we show you how can you convert it into with TF Lite.
• 50:05
  Mahesh: So just three lines of code and you can convert it into TF Lite.
• 50:08
  Mahesh: If you don't want to do that part, it's even simpler.
• 50:12
  Mahesh: You can open it in Colab and then run this lab to convert it.
• 50:16
  Mahesh: But if you don't want to do that, you can actually do much simpler things.
• 50:22
  Mahesh: You can just download it.
• 50:23
  Mahesh: I have put it here, I'll give the link and then what you can do is you just need to change one line of code.
• 50:33
  Mahesh: I put an index file.
• 50:35
  Mahesh: If you download the model, you got a bin file and we have also provided the lab which you can use to convert any other.
• 50:42
  Mahesh: If you don't want to use Gamma, you want to use the stability one to do your images, then just one line code change.
• 50:49
  Mahesh: Once you have the model, you can go to this file in this indexjs file and change the location of the model.
• 50:56
  Mahesh: You can go to this file in this indexjs file and change the location of the model which you just converted and after that just run this command which runs this JavaScript, and then you can open your localhost and start inferencing.
• 51:10
  Mahesh: I did that.
• 51:11
  Mahesh: So here is how it looks and it's running right here on my device and I can show you my terminal.
• 51:19
  Mahesh: So basically all I did is I said this command and before this I just went to, I went into that machine, changed the location to the model that I brought in or converted with TensorFlow Lite, and then I can just ask questions and it will continue to give me responses.
• 51:39
  Mahesh: So just that easy.
• 51:43
  Mahesh: The world we live in is so easy, which is that you can just take any model.
• 51:49
  Mahesh: This is years and years of work of first building a large language model, then distilling it to a small language model, and then open sourcing it on Hugging Face, which you can download in this lab and then convert it into small format which run acceleration with acceleration on your edge devices.
• 52:07
  Mahesh: Edge devices means your phone or your PC.
• 52:10
  Mahesh: And then here's an app which just requires a line code change, and then here's an app which just requires a line code change from the model and then you can run it.
• 52:20
  Mahesh: You can host it as a website if you want to pay five bucks and actually launch something which is local in your company, your machines, your house and don't let your data go away.
• 52:31
  Mahesh: So that's the last thing.
• 52:33
  Mahesh: That's all I had for today.
• 52:38
  Pete: Awesome.
• 52:38
  Pete: We have a couple of more questions coming in and we're going to put links to all this stuff on our Discord.
• 52:43
  Mahesh: Yeah, Let me just put.
• 52:44
  Mahesh: I will put the links somewhere.
• 52:46
  Mahesh: Where should I put comments?
• 52:50
  Pete: We have a bunch of stuff going on on Discord so I encourage people to go over there.
• 52:53
  Pete: Like in the chat.
• 52:54
  Mahesh: So this is the link.
• 52:55
  Mahesh: And, by the way, last thing is, this is my course, that's our community.
• 53:00
  Mahesh: Here is a link to the course.
• 53:01
  Mahesh: Hopefully it opens up one day.
• 53:05
  Mahesh: See, that's what means the things.
• 53:08
  Mahesh: There you go PRADEEP MALLICKI and then I will put the lab course.
• 53:12
  Mahesh: So this is the course, so I can just copy it.
• 53:15
  Mahesh: Comment copy.
• 53:16
  Pete: There was a question that came in around is it common to deploy multiple SLMs that could be swapped on an endpoint?
• 53:25
  Pete: Is that?
• 53:26
  Pete: I mean, we're still early stages, there really aren't a lot of deployments yet, but is that something people are thinking about or is it really sort of like fit for purpose?
• 53:41
  Mahesh: You're going to have an SLM tuned and stuffed into a piece of equipment and that's it.
• 53:43
  Mahesh: Yeah, so I think a couple of things here, right?
• 53:47
  Mahesh: So there are a lot of things people are trying to see how can?
• 53:48
  Mahesh: So now we are living in a world, like the example we took, where you need to have imagine like contract was just one thing in your company and it's the last thing in your company.
• 53:57
  Mahesh: Before that you have much more interesting scenarios, like everybody wants to build their own model, people want to have their own assistant kind of model.
• 54:04
  Mahesh: I want to build my own assistant.
• 54:06
  Mahesh: And if you throw it then you have a lot of model copies that are relying in your organization.
• 54:11
  Mahesh: This is where the idea is going.
• 54:13
  Mahesh: And now if you want to host them at different endpoints, that's cost.
• 54:18
  Mahesh: So how can you have a single or only the delta of the weights and then base weights?
• 54:26
  Mahesh: So if you look at like the equations you will have, you change as you find in these models to your use cases.
• 54:35
  Mahesh: What you can do is this is your base model, that 2 billion parameter model.
• 54:39
  Mahesh: Then you add some adapters to it.
• 54:43
  Mahesh: Let's say, 20 MB worth of adapters.
• 54:45
  Mahesh: This is where you are learning all the things that are specific to your use case.
• 54:50
  Mahesh: And then you are adding this to this 2 billion parameter model.
• 54:54
  Mahesh: So this copy is same, because that's the base model and these adapters are case specific.
• 55:03
  Mahesh: My other marker is not working, so I'm going to just say square, square, square.
• 55:08
  Mahesh: So that's my 20 billion, and then circle, circle, circle is 20 billion, and this is a common practice that is evolving, which is at runtime, let's host just one of the base at runtime, load the adapters that are needed, and that's what I think most of the startups are going after, which is like how can we make this possible?
• 55:33
  Mahesh: So there is this company which is funded by yc uh, I'm forgetting their name, but uh, look at, like fireworks also, fireworksai came up with their inference where they said that, hey, now you can inference at a fraction of cost across your slms and behind the scenes they must be doing something like this.
• 55:52
  Mahesh: And uh, I'm forgetting the name of the company.
• 55:56
  Mahesh: It will come to me that's also doing the same thing.
• 56:00
  Mahesh: Peter Levell, who built this website called AI studio, is using that company.
• 56:08
  Mahesh: He talked about it in the podcast.
• 56:09
  Mahesh: That's where I checked.
• 56:10
  Mahesh: They are using exactly same.
• 56:12
  Mahesh: Where now you have to, everybody will train their own model for their images.
• 56:16
  Mahesh: Let's say I need a photo shot, so I will create my own copy.
• 56:19
  Mahesh: He has a website which does it for like 10 bucks.
• 56:23
  Mahesh: Now he need a copy to be done for everybody.
• 56:25
  Mahesh: How can that scale?
• 56:27
  Mahesh: And that's where he's using this company to scale, and I think that's a scenario that's evolving every day.
• 56:34
  Pete: By the way, I desperately want to get you some whiteboard cleaner, but that's a different conversation.
• 56:44
  Mahesh: I think there's a lot of things that you want me to do.
• 56:51
  Pete: But I really appreciate the time.
• 56:52
  Pete: We're coming to the end of our fun here and I know you need to jump, but, mahesh, great to see you again.
• 56:57
  Pete: You're doing some great work out there educating people, getting people motivated.
• 57:01
  Pete: So we'll put all the links in our Discord server and obviously this recording will be on YouTube and LinkedIn and stuff like that and Twitch.
• 57:08
  Mahesh: Yeah, I'm just posting everything in the comments, so I will stay one more minute just to make sure everything is in the comments, so you can take it from there.
• 57:15
  Mahesh: I'm just pasting it here because I know I go to all these talks.
• 57:19
  Pete: Here's the last question someone's asking.
• 57:21
  Pete: They often give free compression for Onyx.
• 57:25
  Pete: How does it do that quantization or pruning?
• 57:29
  Pete: So how does the Onyx model compression work?
• 57:32
  Pete: If you can answer that in about five seconds, that'd be awesome.
• 57:36
  Mahesh: Onyx models often gives free compression, how does it do quantization and pruning?
• 57:40
  Mahesh: So I think their compression techniques is basically model fusing.
• 57:44
  Mahesh: So what they do is there are a lot of operators that PyTorch will have and they provide an intermediary layer.
• 57:52
  Mahesh: So they say that, hey, instead of using these 10 operators like add, subtract, multiply these are fundamentals.
• 58:01
  Mahesh: And then on top of this there's ReLU6 and ReLU8.
• 58:04
  Mahesh: They look for the best ways to optimize it, because they can first time see the whole graph of your neural network.
• 58:10
  Mahesh: And then they say, instead of these four, I'm going to fuse them, I'm just going to use one.
• 58:14
  Mahesh: And that's the compression they use.
• 58:16
  Mahesh: They don't do quantization or pruning that's something you need to do on before or after but they do compression by doing model.
• 58:24
  Mahesh: Fusing is one of the examples which right now, like PyTorch also tries doing that to do the graph representation that PyTorch 2.0 released.
• 58:33
  Mahesh: But the idea is that if you can look at all the operators, then you can fuse them to reduce the number of operators in your model and if you have, instead of four, you have one, then you're good to go.
• 58:46
  Mahesh: Your size looks one-fourth already and that's your compressed version.
• 58:50
  Mahesh: Awesome.
• 58:53
  Pete: All right, sir.
• 58:53
  Pete: Well, thank you so much, Jenny.
• 58:55
  Pete: Good to see you again, and yeah.
• 58:59
  Mahesh: I will keep tasting the links.
• 59:00
  Mahesh: I don't know why I gave so many links no all right, well, you get people plenty of homework.
• 59:10
  Pete: All right, sounds good.
• 59:11
  Pete: Well, we're going to.
• 59:12
  Pete: We're going to sign out.
• 59:13
  Mahesh: Thanks a lot for having me.
• 59:14
  Mahesh: Thank you, jenny, and I will run to the computer museum next.
• 59:19
  Jenny: Well it's all my colleagues.
• 59:21
  Jenny: I said hi and enjoy.
• 59:23
  Pete: All right, take care everybody, bye, bye.
• 59:26
  Pete: Outro Music.