Latest / Nutanix Weekly / Nutanix Weekly: Running NVIDIA Triton Inference Server on Nutanix

Transcript

Nutanix Weekly: Running NVIDIA Triton Inference Server on Nutanix · Nutanix Weekly

WEBVTT 1 00:00:03.270 --> 00:00:19.899 Philip Sellers: Hello and welcome to another episode of Nutanix Weekly, one of the many XenTegra Podcasts with context this is, Phil sellers, your host. Again for this episode, and wanna say, welcome and thanks for spending a little time with us. 2 00:00:20.701 --> 00:00:32.900 Philip Sellers: We know you probably have very little free time. So while you're on a jog or treadmill, or whatever it is that you're doing listening to us. We we appreciate you spending a little time with us. 3 00:00:33.494 --> 00:00:44.530 Philip Sellers: You know, we we call these things podcasts with contacts because we try to bring the real world experience and the real world conversations. We're having with customers 4 00:00:44.670 --> 00:01:00.339 Philip Sellers: to the conversation here that you're listening to, and, as always, the only way I can do that is, with other great technologists and minds on the phone with us today I've got Mr. Harvey Green. CEO is integr 5 00:01:00.500 --> 00:01:02.800 Philip Sellers: Harvey. How's the world treating you. 6 00:01:03.790 --> 00:01:09.849 Harvey Green III: Pretty good, pretty good. I can't believe that you would imply that people are doing other things while they listen to our podcast. 7 00:01:10.699 --> 00:01:11.549 Philip Sellers: Hey! 8 00:01:11.970 --> 00:01:12.390 Harvey Green III: We're. 9 00:01:12.922 --> 00:01:15.587 Philip Sellers: Multitaskers in this world, right? 10 00:01:17.370 --> 00:01:20.378 Harvey Green III: Yeah, alright, I guess that's true. 11 00:01:21.120 --> 00:01:31.899 Jirah Cox: Are you implying our user base is not like, sitting like seventies like hi-fi headphones, and like the rack Mount Amp, you know, sitting there on the beambag, just totally meditating on just our our only our spoken words. 12 00:01:32.210 --> 00:01:32.870 Harvey Green III: Well. 13 00:01:32.870 --> 00:01:33.329 Philip Sellers: You know. 14 00:01:33.330 --> 00:01:34.360 Harvey Green III: That the way. 15 00:01:34.490 --> 00:01:41.864 Philip Sellers: I've heard my voice. So yeah, I mean, just you know, I I'm not thinking I've got the asmr thing going on. 16 00:01:42.350 --> 00:01:45.650 Philip Sellers: So I'm just appreciative. Somebody's listening at the end of the day. 17 00:01:46.417 --> 00:01:53.402 Philip Sellers: But yeah, I mean other than self deprecating humor. We're also joined with. 18 00:01:53.940 --> 00:02:05.069 Jirah Cox: I bring that to. That's fine. I'm a team player, but no grateful to be a part of the the the Xpu, the Zintra podcast universe. It's great to be here. 19 00:02:06.165 --> 00:02:28.959 Philip Sellers: Well, Jagger, we we all get you joining us. You know, and and bringing the knowledge at the end of the day. That's 1 of the things I really really have enjoyed. It's been over 18 months. I've been here at Zintagra, and I wanna say, I got to meet you week. One cause, I think. Week one. I was on a nutanix podcast with you. 20 00:02:28.960 --> 00:02:31.200 Harvey Green III: You sure were I made sure of it? 21 00:02:33.880 --> 00:02:43.990 Philip Sellers: And and honestly, I mean, it's it's great. Learn so much. And I think everybody listening has learned so much, so just really appreciate you. What you bring to the table? 22 00:02:45.180 --> 00:02:52.580 Philip Sellers: And and speaking of today, we want to talk a little bit about Nvidia on Nutanix. And so 23 00:02:53.060 --> 00:03:20.730 Philip Sellers: bringing to the table, we've got a great blog post from the Nutanix dot dev blog for everybody watching on Youtube replays you can see it on screen. But the title of the blog is running in video. Triton, Interface Server on Nutanix and this is written by Laura Giordano, technical marketing engineering director at Nutanix. So shout out to Laura, thanks so much for posting this and giving us something to talk about today. 24 00:03:21.620 --> 00:03:37.739 Jirah Cox: Fantastic one of the one of the 1st Esses at Utanics and then now a huge player in tech marketing for us. And if you've been to a dot next conference, or one of the bigger dot next on Tours, you've actually probably seen Laura in person. Usually, you know. 25 00:03:38.290 --> 00:03:45.880 Jirah Cox: running some awesome, awesome Demos right there from stage of something that like just gets announced. And like 2 min later, you see Laura giving a demo of it, it's awesome. 26 00:03:46.550 --> 00:03:52.645 Philip Sellers: That's awesome shout out to Laura, thanks so much for giving us fun stuff to talk about. 27 00:03:53.310 --> 00:04:00.329 Philip Sellers: So I mean, this is big news for for Nutanix and and across the space right doing 28 00:04:02.060 --> 00:04:06.300 Philip Sellers: doing AI AI type things on prem. 29 00:04:06.856 --> 00:04:14.190 Philip Sellers: this was a huge topic. As we were out at New tanks next a few months ago in Barcelona. 30 00:04:14.910 --> 00:04:16.110 Philip Sellers: I mean 31 00:04:16.339 --> 00:04:25.569 Philip Sellers: a as you guys have thought about this. I mean, what are the used cases, or what are the things you've had conversations following up of of on prem AI. 32 00:04:26.850 --> 00:04:28.170 Jirah Cox: Oh, we have, I mean. 33 00:04:28.310 --> 00:04:30.150 Jirah Cox: huge and and 34 00:04:30.170 --> 00:04:46.199 Jirah Cox: pretty lengthy, right? But some, of course, some of the top ones. I even have customers that already are running things like on prem copilot right to get help. Their software developers write code faster and better, but of course, fully, privately, and of course, with performance. 35 00:04:46.670 --> 00:04:52.419 Jirah Cox: any kind of thing stuff like inference, image detection computer vision 36 00:04:52.570 --> 00:04:58.710 Jirah Cox: chat bots. Of course, we've actually launched a number of them, even internally, here at Nutanix. Right? That we're sort of cutting our teeth on around. 37 00:04:58.720 --> 00:05:11.190 Jirah Cox: you know. Help, query, knowledge bases get answers faster. Help, support customer service kind of outcomes. So yeah, it's the the list is almost limited only by your imagination of what it could help. 38 00:05:11.310 --> 00:05:12.480 Jirah Cox: You do 39 00:05:12.740 --> 00:05:16.150 Jirah Cox: right. Usually there's more questions around, how do I start doing it? 40 00:05:16.420 --> 00:05:17.080 Philip Sellers: Yeah. 41 00:05:17.580 --> 00:05:22.160 Philip Sellers: yeah. Harvey, you and I, I know, spent some time doing hands on labs 42 00:05:22.490 --> 00:05:26.248 Philip Sellers: trying to understand. How. How do you do that I mean. 43 00:05:26.850 --> 00:05:34.679 Philip Sellers: You know what what kind of stands out to you as as customers start to have those conversations. And the how do you do it? 44 00:05:35.840 --> 00:05:37.991 Harvey Green III: Yeah, I mean, I think that 45 00:05:38.780 --> 00:05:52.240 Harvey Green III: there's a lot of talk about this overall. And then when you get past the overall talk and we start talking about. Should we do it and get past that into now, how do we do it? 46 00:05:53.055 --> 00:05:54.690 Harvey Green III: You know the 47 00:05:54.710 --> 00:05:59.060 Harvey Green III: the picture gets deeper and deeper each time. But 48 00:05:59.606 --> 00:06:18.360 Harvey Green III: this is this covers one of those top level decisions of, are we going to do this internally and keep our data here or externally, and use, you know, something that's already fabricated in there. This gives you the option to do that internally. 49 00:06:18.450 --> 00:06:26.270 Harvey Green III: Keep your data where it is maintain. You know, the chain of custody over that. So it's not going anywhere. 50 00:06:26.670 --> 00:06:32.790 Harvey Green III: And I'm I'm sure that makes security people just a little bit happier. In some cases. 51 00:06:33.770 --> 00:06:35.260 Philip Sellers: I think, you know. 52 00:06:35.410 --> 00:06:44.549 Philip Sellers: tailing on to what you just said, there's certain data, use cases where we could never, that that data was never leaving our data center so we couldn't tap into. 53 00:06:44.550 --> 00:06:45.210 Harvey Green III: Then. 54 00:06:46.090 --> 00:06:51.350 Philip Sellers: We couldn't tap into that. And and we couldn't do things on that. So that definitely 55 00:06:51.360 --> 00:06:54.189 Philip Sellers: opens new use cases in and of itself. 56 00:06:54.660 --> 00:06:58.869 Harvey Green III: Yeah, I mean, one of the use cases you've already brought up is, you know, being able to 57 00:06:59.140 --> 00:07:03.619 Harvey Green III: go through that internally and be able to help it. 58 00:07:04.275 --> 00:07:07.428 Harvey Green III: Help the developers write better code. 59 00:07:09.170 --> 00:07:13.040 Harvey Green III: that in and of itself you've got developers developing 60 00:07:13.080 --> 00:07:17.416 Harvey Green III: against something that is not out there, not, you know, 61 00:07:18.260 --> 00:07:22.629 Harvey Green III: not out to users, not, you know, generally available. 62 00:07:23.094 --> 00:07:27.215 Harvey Green III: That you're still doing testing on that. I mean, definitely has 63 00:07:27.690 --> 00:07:32.670 Harvey Green III: some some business intelligence in there. You know some some of the 64 00:07:33.277 --> 00:07:40.820 Harvey Green III: intellectual property that you want to keep safe while you're still in the process of actually making it something 65 00:07:41.955 --> 00:07:46.210 Harvey Green III: and keeping that in house is very important. 66 00:07:48.340 --> 00:08:00.070 Philip Sellers: I, I 100% agree so tactically, one of the ways that we can do this now is is tapping into the Nvidia Gpus in cluster. You know you and I both 67 00:08:00.490 --> 00:08:08.740 Philip Sellers: worked on some kubernetes, or I I guess it was technically a Mini K 8 cluster, just a kind of a small scale 68 00:08:09.276 --> 00:08:19.499 Philip Sellers: doing labs during dot next. And so a lot of this stuff is written in Cloud native. It's it's already released as containerized workloads. 69 00:08:19.520 --> 00:08:30.029 Philip Sellers: So you gotta have a basis in that. And then, or not necessarily. In this case, you know, the article we're talking about is is creating a Gpu enabled Vm. 70 00:08:30.507 --> 00:08:43.609 Philip Sellers: but then, basically, you, you take some open source projects or some commercial software. And you you bring that all together. So let's kind of talk a little bit through this one. And 71 00:08:44.041 --> 00:08:56.260 Philip Sellers: it. It's very, I'm gonna say, cut and dry. It's a step by step build process for running the Nvidia inference server, the Triton inference server on prem. 72 00:08:56.870 --> 00:08:57.530 Philip Sellers: But 73 00:08:58.410 --> 00:09:05.629 Philip Sellers: I I guess the introduction walks us through a couple of things. Key. Announce things like that, Jair, do you wanna kinda set the stage for us. 74 00:09:05.920 --> 00:09:24.471 Jirah Cox: Totally right? So this is is gonna be to your point, Philip, like a very prescriptive walkthrough like, do this, do this, do this, and then at the end of the cooking show, you're gonna have a similar kind of a you know roast turkey in this case a a Triton inference server. The what this brings together right is, of course, the awesome 75 00:09:24.770 --> 00:09:37.009 Jirah Cox: platform, right? How do I deploy vms as an automated service, a substrate that I need like a gpu like you mentioned. Of course, being in video certified great ingredient for a wonderful recipe there. 76 00:09:37.480 --> 00:10:01.129 Jirah Cox: so having nutanix nodes that are on the Nvidia certified systems list. And of course, lastly, something to do right? So this is gonna show deploying, hugging face. Who actually, more recently than this article, right? As we got like, we also learned together in Barcelona, New Nutanix technology partner, right? So you can now totally get wonderful all the way joint support from Nutanix and hugging face for deploying things like this. 77 00:10:01.738 --> 00:10:05.223 Jirah Cox: And of course, the last thing to probably say that 78 00:10:06.246 --> 00:10:22.840 Jirah Cox: just being a good corporate citizen. This is, of course, one way to deploy AI models onto Nutanix. We actually announced@our.next show right coming soon that Nutanix gpt in a box will be will be an even easier, like more of a more of a product than a process. Right? Click, click next finish. 79 00:10:22.950 --> 00:10:24.840 Jirah Cox: Now you're running AI in production 80 00:10:25.050 --> 00:10:34.789 Jirah Cox: style of outcome. This is more of a pop. Let's pop the hood. Let's do it from the Cli and get that all kind of running in a bit more of a manual, controllable fashion as well. 81 00:10:35.540 --> 00:10:41.060 Philip Sellers: Yeah. And I think that's important. Like, Gpt in a box came out over a year ago. Version one 82 00:10:41.080 --> 00:11:03.739 Philip Sellers: Gpt. Innovation 2 was announced during dot. Next, moving it more from a project product and helping users adopt it easily with workflows and engines to to get value out of it more quickly, and hugging face. As you said. You know it's a community with all the tools and things to help you with your AI development. So 83 00:11:03.740 --> 00:11:18.210 Philip Sellers: it's bring that side of the equation, the development, the the software side of it to to what we need. So probably not going to need to go into depth as we go through this, but I mean. 84 00:11:18.210 --> 00:11:19.730 Jirah Cox: Principles will still apply. 85 00:11:19.860 --> 00:11:21.213 Philip Sellers: Yeah. Principles. 86 00:11:22.100 --> 00:11:29.640 Philip Sellers: first, st step up. We're going to create a Gpu enabled. Vm, obviously, you've got to be able to consume the Gpu cycles. So 87 00:11:29.710 --> 00:11:32.598 Philip Sellers: that makes a lot of sense. But. 88 00:11:32.960 --> 00:11:51.290 Jirah Cox: Cool 2 neat takeaways here with the way Laura does this, that you know, especially if you're coming to New Tan X, and you have more of like what I used to have a very traditional windows. Server admin background couple of really cool technologies here. One is that a lot of your Linux vendors. In this case we're gonna show using ubuntu in the late Lts. There, 22 0, 4. 89 00:11:51.330 --> 00:12:07.820 Jirah Cox: The Linux vendors will give you. Here is a full on bootable, virtual disk image. It's not an iso for installation. It's a virtual CD virtual hard drive. You can just power it on right just download it, turn it on one of the rap most rapid ways to get a workload up and running right just download a Linux image. 90 00:12:07.860 --> 00:12:23.439 Jirah Cox: Second one, though, is showing the cloud in it file, which is a way to as you're creating the Vm. Paste in a little block of text that says, essentially like an on 1st boot. Personalize the Vm. This way. So in this case, Laura's doing that to turn on Ssh. 91 00:12:23.440 --> 00:12:44.219 Jirah Cox: so the Vm. Can then be accessed over Ssh. By a putty, or the windows command line or Mac command line, and secondly, set a password for that. So now we can do that do that securely as well. So some really neat tricks to take away here from just a general. Sys admin point of view of ways to get Linux up and running on Nutanix rapidly. Whether we do a Gpu or not, whether we're doing AI or not. 92 00:12:45.180 --> 00:12:53.492 Philip Sellers: Yeah, absolutely. You know, the key here is that that pass through. That's what we're tapping into in the virtual machine. 93 00:12:54.980 --> 00:12:56.400 Philip Sellers: But the guest system. 94 00:12:56.400 --> 00:12:59.309 Jirah Cox: Have a Gpu step one, pass it through. 95 00:13:01.510 --> 00:13:09.719 Philip Sellers: Yeah. And then again, like, you're pointing out cloud in it. The scripting language to just onboard that make it easier to 96 00:13:09.850 --> 00:13:13.159 Philip Sellers: to do this in mass, right? I mean. 97 00:13:13.450 --> 00:13:32.369 Philip Sellers: automation has always been a core component of nutanix. Being able to pass that through onto the software layers is now a critical success factor as you try to do projects like AI inference engines on the infrastructure. So just following through that same thread of 98 00:13:33.840 --> 00:13:42.160 Philip Sellers: heavy, I'm gonna say heavy but I don't mean heavy in a bad way. I mean heavy in a good way. Very orchestrated workflows. 99 00:13:43.150 --> 00:13:48.230 Jirah Cox: Yeah, that's a good point, like, like thinking about like zooming in from like the house to like a brick 100 00:13:48.280 --> 00:13:53.170 Jirah Cox: like this could be a work flow where, like my Vm creation, automation might 101 00:13:53.550 --> 00:14:10.350 Jirah Cox: randomly generate that password and stash it in a vault or an itsm versus. In this case, use it as hard coded right, but it shows where you could stick that variable in and use this for a a more repeatable, more broad more secure automated process. 102 00:14:10.520 --> 00:14:10.930 Philip Sellers: Yeah. 103 00:14:11.490 --> 00:14:24.320 Philip Sellers: yeah, it looks like in this example, there's still some manual stuff that we have to do once again created nothing going. We have to install some drivers. It has to know how to talk to the Nidia Gpu at the end of the day. 104 00:14:26.180 --> 00:14:33.039 Philip Sellers: So what else are we doing here? It looks like we. We install some drivers, and then what's next? 105 00:14:34.400 --> 00:14:41.120 Jirah Cox: We're installing drivers, verifying the output of that, of course, installing docker right? When I have a way to to run containerize applications. 106 00:14:41.350 --> 00:14:47.650 Jirah Cox: So installing docker, installing the Nvidia container toolkit. That shows that coming out of the Nvidia Github 107 00:14:47.820 --> 00:14:49.310 Jirah Cox: Repo, there. 108 00:14:50.400 --> 00:15:02.209 Philip Sellers: And that's that's something that's stands out to me like we're going after repos and pulling this stuff. We're not writing this stuff, but we're just tapping in and pulling it down. We're consuming it. At the end of the day. 109 00:15:02.610 --> 00:15:03.720 Jirah Cox: Yeah, totally true. 110 00:15:06.012 --> 00:15:06.785 Jirah Cox: The 111 00:15:08.860 --> 00:15:29.029 Jirah Cox: installing mini conda small, which I just checked out. They're good their repo as well to see what that is, Mini Conda is a free minimal installer for Conda, which is a Bootstrap version of Anaconda that includes Conda Python. Other things like that Pip. C. Lib, and a few others. So application, kind of framework runtime, to get python up and running in a kind of predictable state. There. 112 00:15:29.230 --> 00:15:29.950 Philip Sellers: Okay. 113 00:15:30.550 --> 00:15:32.100 Jirah Cox: Selling, many, many condo. 114 00:15:32.100 --> 00:15:34.194 Philip Sellers: What's Minicanda? So 115 00:15:36.560 --> 00:15:42.489 Philip Sellers: you know, as an infrastructure professional coming into this, there's so many different 116 00:15:43.242 --> 00:16:03.890 Philip Sellers: tools, libraries, things like that, just understanding all the different components that we're tapping into is sometimes a difficulty as you approach these projects. But I think, that's where a quick start like this definitely helps us to to see it and then start to build on what we've successfully done to do. The next thing. 117 00:16:04.080 --> 00:16:11.440 Jirah Cox: Totally. In some ways, I think you know, standing up AI or Gpt or Llm based solutions in a data center are. 118 00:16:11.894 --> 00:16:38.110 Jirah Cox: maybe the most hardware defined thing that some of us have done in 20 years. Right? We've been doing things in vms, right things where it's just like, define your resources in an abstract way of CPU memory storage and this is kind of getting back more, very close to the metal of like no, you. D! Thou shalt have this Gpu which needs this driver to talk to it. And here's how you how you stand that up. Kind of stuff much more prescriptive than some so than what some of us are very used to for our day jobs. 119 00:16:39.659 --> 00:17:06.910 Jirah Cox: So getting that the Trident Inference Service stood up itself again. Here again we see that that pulls down from Github. The inference server is, of course, like Nvidia's packaged way of running this kind of as a larger, like framework of like, how can I run models and Gpt's and Llms in my data center? I could run tens of thousands of them, and then I could do training. I could run one or 2, or 3, and then I could do inference, or run an app or do a proof of concept. 120 00:17:07.274 --> 00:17:11.960 Jirah Cox: So this, of course, is going to be a single node. But there's no reasons. Couldn't be scalable as well. 121 00:17:12.599 --> 00:17:17.809 Jirah Cox: So pretty simple. There, get clone, and then run a script that it provides there locally. There. 122 00:17:18.050 --> 00:17:18.359 Philip Sellers: Yeah. 123 00:17:18.369 --> 00:17:31.189 Jirah Cox: And then, of course, we launch it as a as a docker run right? So here we see that docker framework standing up the containerized micro service that it provides there to run the Triton server and expose that on a web server. 124 00:17:31.610 --> 00:17:55.840 Philip Sellers: Yeah. And so I, I think it's important. You know, as we talk about this, we're looking at Docker. This is a proof of concept. This is not a production scale thing. Likely the same container would run on a Kubernetes infrastructure at scale with other things configured. This is more of a Poc type proving ground where you can easily get things up and running. But this is a 125 00:17:55.910 --> 00:17:57.060 Philip Sellers: a single 126 00:17:57.090 --> 00:18:01.200 Philip Sellers: instance sort of thing, not not something that's ready for production. 127 00:18:03.690 --> 00:18:06.778 Philip Sellers: So we've got a talker container up. 128 00:18:07.740 --> 00:18:11.070 Philip Sellers: you know. Then we send an inference request. 129 00:18:12.790 --> 00:18:13.869 Philip Sellers: What is that. 130 00:18:15.120 --> 00:18:20.199 Jirah Cox: The inference request. I'm catching up with you here as well is 131 00:18:20.650 --> 00:18:27.379 Jirah Cox: actually passing. It looks like a Jpeg here that's asking it to to infer right 132 00:18:27.840 --> 00:18:50.040 Jirah Cox: spoiler inference. Server infers. What's in this picture? I just gave you right. And what's really neat is so of course this is. This is to your point, Philip. Very proof of concept. Ish very kind of not quite rudimentary, but very low level. Right? I give it an image. I get back text about what's in the image right? And like confidence scores. I even. I think that's what the numbers there imply 133 00:18:50.040 --> 00:18:58.790 Jirah Cox: coffee, mug, cup coffee pot, right? What was in what I just gave you? So image class image classification type activities here. 134 00:18:59.170 --> 00:19:01.650 Philip Sellers: So I'm scrolling back up a little bit so like 135 00:19:01.690 --> 00:19:03.140 Philip Sellers: that. 136 00:19:03.840 --> 00:19:09.500 Philip Sellers: That seems like magic to me right? I've just pulled a couple of things from Github. I've I've 137 00:19:09.640 --> 00:19:14.100 Philip Sellers: says spell. And suddenly it's giving me this text. It says it's coughed up. 138 00:19:14.460 --> 00:19:21.769 Philip Sellers: But that's not actually what's going on above. When we launch this, we and I think we, we 139 00:19:21.830 --> 00:19:36.440 Philip Sellers: may skipped over this, but we fetched a model. And and I think that's an important concept for people to talk about when we talk about AI is that there's a model that we pulled as well. That model was trained somewhere else. 140 00:19:36.670 --> 00:19:43.960 Philip Sellers: That model is being consumed. But we're benefiting from the knowledge of that model that 141 00:19:44.170 --> 00:19:47.451 Philip Sellers: that happened. That training happened somewhere else. 142 00:19:48.431 --> 00:19:51.199 Philip Sellers: and so it's not black magic. 143 00:19:51.390 --> 00:19:55.899 Philip Sellers: Somebody spent a lot of CPU cycles creating that model that we're now consuming. 144 00:19:56.090 --> 00:19:56.420 Harvey Green III: Yeah. 145 00:19:56.420 --> 00:19:57.150 Jirah Cox: Oh, wait! 146 00:19:57.150 --> 00:20:10.050 Harvey Green III: I kinda liken this to encyclopedias only because I I tell people all the time my kids will never know what it's like to pull an encyclopedia off the shelf and have to look something up. 147 00:20:10.808 --> 00:20:16.730 Harvey Green III: Th. This is like bringing in the encyclopedia and basically teaching 148 00:20:16.940 --> 00:20:23.119 Harvey Green III: what you needed to know in order to accomplish the things that you're planning on doing in the near future. 149 00:20:23.510 --> 00:20:24.130 Philip Sellers: Yeah. 150 00:20:24.680 --> 00:20:25.993 Jirah Cox: Totally the 151 00:20:27.430 --> 00:20:29.769 Jirah Cox: Yeah. Fully agree with you, Harvey. Like, if you're listening. 152 00:20:29.770 --> 00:20:31.840 Harvey Green III: May have dated myself there. No. 153 00:20:31.840 --> 00:20:32.370 Jirah Cox: Well, that? 154 00:20:32.750 --> 00:20:43.599 Jirah Cox: Well, the the phone, the phone version I'm doing to my kids lately, they're 8 and 11 is like, if we're driving along. I'll point to like a telephone pole and go kids. What do you think that is? And why does it exist? And what does it do. 155 00:20:43.770 --> 00:20:45.259 Harvey Green III: Right, God. 156 00:20:45.390 --> 00:20:49.660 Jirah Cox: It is just not part of their life in any degree whatsoever. 157 00:20:49.840 --> 00:20:50.240 Harvey Green III: Right. 158 00:20:50.640 --> 00:20:59.149 Philip Sellers: Not to completely take us in a rabbit hole, but we call them telephone calls, but they run electric wires. They don't run telephone wires for the most part anymore. Right? 159 00:20:59.180 --> 00:21:06.059 Philip Sellers: I mean, I I that's a complete. They're both looking at me. 160 00:21:07.520 --> 00:21:12.780 Jirah Cox: The I wouldn't say, Yeah, they're not used very much. But no, you're you're right, Harvey. If you're listening to this, like 161 00:21:12.830 --> 00:21:22.950 Jirah Cox: the vast majority of customers like, if you are a customer who as as part of running it, you consume power, you don't generate power. 162 00:21:22.960 --> 00:21:31.809 Jirah Cox: You're probably gonna consume AI models. You're not gonna train AI models. Right? Same kind of thing is, it is done elsewhere. And you get the benefit of it within your 4 walls. 163 00:21:31.960 --> 00:21:32.580 Jirah Cox: Yeah. 164 00:21:32.580 --> 00:21:39.529 Philip Sellers: So people talk about the energy consumption of AI, right? You know. 165 00:21:39.780 --> 00:21:41.920 Philip Sellers: that's not what this is. 166 00:21:41.930 --> 00:21:45.689 Philip Sellers: That's the other thing. That's the training. That's where 167 00:21:45.750 --> 00:21:52.229 Philip Sellers: these models take enormous amounts of power and processing to to be created. 168 00:21:52.510 --> 00:22:00.450 Jirah Cox: And maybe literally dozens of thousands of Gpus, all working in parallel for weeks or months on end. To do that training, activity. 169 00:22:00.630 --> 00:22:04.555 Philip Sellers: Yeah. And and I think that's a a really important call out, 170 00:22:05.180 --> 00:22:09.440 Philip Sellers: because, you know, it's not just happening in the box. But 171 00:22:09.610 --> 00:22:15.829 Philip Sellers: there are things certainly happening in the box that we're talking about. It's just not training the model. 172 00:22:17.483 --> 00:22:18.130 Philip Sellers: So 173 00:22:18.620 --> 00:22:34.419 Philip Sellers: then we close this out. This example of the coffee cup, and then we move on to another example loading a hugging face model in trident. So now we're gonna use another model loaded into the trident engine. What? What comes next? 174 00:22:35.500 --> 00:23:00.159 Jirah Cox: Yeah. So now that we've sort of tested like we, we have proof of life. Right? We've done like a hello world like that's neat. I can do that one time as a command line. Standing it up. What can I do? That's more business useful. So this shows deploying hugging face itself as a model into trident, right? So there's gonna be one thing now that the trident server like vens to the business or for our Dev teams to go consume 175 00:23:02.350 --> 00:23:10.360 Jirah Cox: So we have the text block here that by a python stands up and deploys a hugging face into 176 00:23:10.724 --> 00:23:14.139 Jirah Cox: this is, of course, using that conda that we previously installed there 177 00:23:14.390 --> 00:23:20.699 Jirah Cox: that ultimately gives me a python based pipeline that I can then use for. 178 00:23:21.220 --> 00:23:34.249 Jirah Cox: This is showing an example where it actually does it via text, right? So like, give me some text generation here, where, if I provide it with like an AI, you know. Usually the word is prompt of my favorite food is. 179 00:23:34.450 --> 00:23:35.570 Jirah Cox: and just let it 180 00:23:35.620 --> 00:23:50.302 Jirah Cox: riff right? What does it think like a lot of take a a guess with 0 data whatsoever what your favorite food is. And we see here the sample return is like iced coffee and chocolate, which actually, it's we're so far pretty pretty accurate. This all sounds amazing. 181 00:23:51.256 --> 00:24:11.313 Jirah Cox: and almond sugar, I mean, yeah, that's pretty neat. So so this one is showing a text based. Use case of when I pass it. Prompt finish the prompt for me. Right? So you can picture that working with Kv articles, or, you know, user provided questions or answers, or what's the way to configure this or whatever? What's our sop for a given activity? 182 00:24:18.795 --> 00:24:19.170 Harvey Green III: Stuff. 183 00:24:20.340 --> 00:24:21.739 Jirah Cox: So then, beyond that. 184 00:24:21.780 --> 00:24:27.772 Jirah Cox: this is showing ways to store multiple models right? Like hugging face like others 185 00:24:28.360 --> 00:24:30.360 Jirah Cox: elsewhere. For 186 00:24:30.540 --> 00:24:31.730 Jirah Cox: the 187 00:24:32.010 --> 00:24:35.599 Jirah Cox: inference engine to then go consume. Right? So this is where 188 00:24:36.100 --> 00:24:59.023 Jirah Cox: we've said it before on the podcast, but worth it worth reiterating. When you run things in, say, like a container has environment. The containers themselves are, of course, fully immutable and don't store data within them. So you'd be putting data that you want to infer on or storing models somewhere else in some other directory. Right? You can do it locally on the Vm. You can do it on an Nas. Nfs. Smb, you can do it on 189 00:24:59.490 --> 00:25:06.999 Jirah Cox: You could do it on other storage. You could do it on block storage. Lots of ways you could solve for that. So this is showing other ways. I can configure a repository 190 00:25:07.050 --> 00:25:16.980 Jirah Cox: to. And how do I set up my directory structure there, and my indexing and and naming or nomenclature to get that aligned the way that the application expects. 191 00:25:18.186 --> 00:25:23.049 Jirah Cox: and then finally, our model code right to say, How do I run different models there. 192 00:25:24.230 --> 00:25:34.290 Jirah Cox: within the inference. Engine talks about importing required libraries right around Triton, Python backend utils works required for every Triton Python model 193 00:25:34.830 --> 00:25:44.676 Jirah Cox: calls out numpy specific python library there, that does multi dimensional array computing and then transformers. That's library that or that provides hugging face. 194 00:25:45.120 --> 00:25:50.190 Jirah Cox: and also sports, you know, downloading and training models itself. Right transformers is like one form of a 195 00:25:50.200 --> 00:25:54.759 Jirah Cox: of an Llm usually text base. I think from what my understanding. 196 00:25:58.500 --> 00:26:05.421 Philip Sellers: A lot of the code that we're looking at on screen, and I know that folks re listening can't really see this. But 197 00:26:05.710 --> 00:26:06.979 Harvey Green III: I'll read it to all. 198 00:26:10.682 --> 00:26:12.800 Philip Sellers: I was. Gonna say it is. 199 00:26:12.800 --> 00:26:14.770 Jirah Cox: Paid on episode length. Yeah. 200 00:26:15.720 --> 00:26:19.619 Philip Sellers: Pretty human readable, though I mean, you know, it's 201 00:26:19.770 --> 00:26:21.850 Philip Sellers: it's approachable. 202 00:26:22.060 --> 00:26:23.659 Philip Sellers: Yes, and yes. 203 00:26:23.660 --> 00:26:31.849 Jirah Cox: What this is translating is what we did before was like a single Linux command line of like, look at this, Jpeg, give me, tell me what's in it, and coffee cup and whatnot. 204 00:26:32.090 --> 00:26:48.250 Jirah Cox: Let's evolve that into what can I consume, you know, securely over, say, like a web, call right? Or an Api call more scalable for the enterprise. Right? Doesn't need to be at the command line of the Linux Vm, but ultimately kind of same output right? I wanna I wanna pass an input to an Llm and get the response back out of it. 205 00:26:48.570 --> 00:27:11.990 Philip Sellers: In in as a consumer. So you know it. Professionals, infrastructure professionals. We all consume applications all day long. Whether it's sas for our. I don't know time cards or our payroll, or, you know, service. Now for our it tickets and things like that. We're all consumers. So what actually is being done here 206 00:27:12.130 --> 00:27:30.449 Philip Sellers: to Gyra's point is more of a Api call type thing, a lot of these engines and services. They're exposed, as Api calls, so that other services can interact with them. And I think that's that's another important thing is is the power of this is underneath the covers. 207 00:27:30.450 --> 00:27:44.420 Philip Sellers: But there has to be that interface. And it's not gonna be just a generic web page like we're accustomed to coming to, you know, to interact and do things that may then talk to this on the back end. But a lot of these 208 00:27:44.440 --> 00:27:56.779 Philip Sellers: models then get used and consumed through other web services through the Api. So yeah, I mean, all this stuff sounds neat. And I think we 209 00:27:57.000 --> 00:28:05.235 Philip Sellers: we may incorrectly think that this is chat gpt. And there's gonna be a nice web interface, and we type in a box, and off we go. 210 00:28:05.960 --> 00:28:10.709 Philip Sellers: That's just one way of interacting with this, and more than likely 211 00:28:10.820 --> 00:28:16.580 Philip Sellers: in the enterprise. It'll use an Api from other other places to actually talk to 212 00:28:16.790 --> 00:28:22.970 Philip Sellers: what we're doing with these inference servers and services that we deploy in the enterprise. 213 00:28:23.200 --> 00:28:23.860 Harvey Green III: Yes, we. 214 00:28:23.860 --> 00:28:24.760 Jirah Cox: Yeah, and I can picture. 215 00:28:24.760 --> 00:28:28.685 Harvey Green III: And it. There's a command line option, and there's a gui option. 216 00:28:29.340 --> 00:28:46.980 Jirah Cox: Yeah, I can picture most of our customers right? If we walked in with this tomorrow, you know, the 1st question would get is, can you make this connect to teams or slack or whatnot? So you know, not a web server, but like, put it in the application that I'm already using. So that kind of connectivity right? Is kind of that. Usually that last mile dial tone. If you will. 217 00:28:47.750 --> 00:28:52.210 Philip Sellers: And and I think that's that's the key takeaway. You know. 218 00:28:53.090 --> 00:28:58.629 Philip Sellers: we we have to think about not only what happens in the processing. But how do we interface with 219 00:28:58.870 --> 00:29:02.510 Philip Sellers: with these kind of services moving forward? And I I 220 00:29:02.640 --> 00:29:19.450 Philip Sellers: I think you hit the nail on the head. We want to consume it where we already are, because I need another channel to check like a hole in my head. I've already got teams and slack and email and text messages everything else. We just wanna consume it where we already are. 221 00:29:22.600 --> 00:29:26.650 Philip Sellers: so this is a great model. And a great example. 222 00:29:27.120 --> 00:29:34.830 Philip Sellers: This is inference focused. This is Gpt, it's that generative AI sort of stuff. 223 00:29:36.440 --> 00:29:47.152 Philip Sellers: through the examples we've looked at. You know it. It's taking image data or things like that helping us understand it, pull context out of out of these data sets. 224 00:29:47.860 --> 00:29:48.750 Philip Sellers: what? 225 00:29:49.410 --> 00:29:51.780 Philip Sellers: I guess. What are? 226 00:29:53.580 --> 00:29:57.209 Philip Sellers: I guess taking something like this? What would be the next step to 227 00:29:57.490 --> 00:30:02.250 Philip Sellers: to take it out of proof of concept and and move forward for the enterprise. 228 00:30:03.540 --> 00:30:19.170 Jirah Cox: I would say, look, having kind of read through the blog post, as we've described it to our audio. Only listeners. This one's worth this one's 1 that's worth clicking through into. Right? Find this article on, you know, running running in video trident inference server on Nutanix. It should be the 1st Google hit 229 00:30:19.460 --> 00:30:25.950 Jirah Cox: for that. But when you do this like this is actually really accessible, it's easy to get up and running in a lab environment in a home lab environment. 230 00:30:25.990 --> 00:30:27.670 Jirah Cox: You're looking at less than 231 00:30:28.040 --> 00:30:43.044 Jirah Cox: 150 ish lines of text, maybe under 100. All of it's, you know, thoroughly copy pastable, so easy to grasp. Really good way to cut your teeth on this. Beyond that, I think for most it professionals. It's 232 00:30:44.394 --> 00:31:05.729 Jirah Cox: partnering with the business right to understand like, where are we looking to use this? Where do we want to consume, you know fully Gpt. Lms in the cloud where there's not sensitive data in play. Where do we need to have a strategy around private performant? AI that lives only behind our firewall, where we do want to have it touching things that we for data, we would don't want leaving our 4 walls. 233 00:31:05.980 --> 00:31:06.800 Jirah Cox: Yeah. 234 00:31:07.450 --> 00:31:10.419 Philip Sellers: And then, you know, is it is it consumable. 235 00:31:10.620 --> 00:31:14.800 Philip Sellers: you know, or do we need to partner with a 3rd party? 236 00:31:15.190 --> 00:31:19.260 Philip Sellers: I I would. I would add that one to it. So there's lots of different 237 00:31:20.650 --> 00:31:48.239 Philip Sellers: technologies and ISP softwares out there. So if you're trying to do something with video, maybe you know safety. Maybe safety is your core. Use case there, and you're trying to spot bad things in video feeds. You know, an inferencing engine like this could be a big part, play a huge role in that. But it may need layers of software developed on top of it to be able to do what you're looking to solve for so the 3rd party markets, probably 238 00:31:48.400 --> 00:31:49.990 Philip Sellers: a big 239 00:31:50.000 --> 00:31:53.200 Philip Sellers: component in a lot of these cases, too. 240 00:31:55.650 --> 00:32:09.209 Philip Sellers: closing thoughts, guys, I mean this, this is interesting. It's it's cool to geek out and and kind of talk through this. It's really cool that we have the capability. I think that's more than anything for me. What? What's your key? Takeaways? 241 00:32:11.650 --> 00:32:21.749 Jirah Cox: Yeah, I think it's I think it's really neat. How democratizing the technology is right, like, never before have you been able to do so much with kind of like, so little like. I just have an old Gp laying around. I can go play with this tomorrow. 242 00:32:21.970 --> 00:32:22.600 Philip Sellers: Yeah. 243 00:32:24.090 --> 00:32:31.699 Harvey Green III: Yeah. And I think being able to have a model like this that's approachable on a platform you're already using. 244 00:32:32.055 --> 00:32:46.260 Harvey Green III: Just something you can go in for for the audio. Only people. They at least have to come in here and look at some code, since Phil and Jyra wouldn't let me read it. But it's it's very approachable. It's something that 245 00:32:46.350 --> 00:32:50.704 Harvey Green III: you can do and actually see the end of it. 246 00:32:51.610 --> 00:33:00.869 Harvey Green III: you know, within a very short period of time I feel talked about. We we did it, you know, in a in a hands-on lab. So you know. 247 00:33:00.970 --> 00:33:08.340 Harvey Green III: you can see the end and see the results at the end of it within a day, in some cases. So 248 00:33:08.530 --> 00:33:11.140 Harvey Green III: I definitely have the 249 00:33:11.420 --> 00:33:13.559 Harvey Green III: the power and the ability 250 00:33:13.891 --> 00:33:17.938 Harvey Green III: to get something like this going internally. Take a look at it, and 251 00:33:18.440 --> 00:33:20.811 Harvey Green III: I'll just put all your hugging face. 252 00:33:21.564 --> 00:33:38.965 Philip Sellers: Right. That's that's a great way to bring it back around and and now for all of our sleep challenge listeners we are gonna take to the next section of our podcast episode and Harvey's gonna read, you code. So. 253 00:33:39.380 --> 00:33:41.290 Jirah Cox: The medication. That's our tagline. 254 00:33:41.290 --> 00:33:48.171 Harvey Green III: I'm gonna need you to help me with some of these words. I can't pronounce. 255 00:33:48.880 --> 00:34:11.060 Philip Sellers: Well, guys, a as always appreciate you guys joining us talking through capabilities in the Nutanix platform. And you know, at the end of the day. It is a platform, right, all of these capabilities, all in the same style of operating heavily orchestrated, heavily workflowed. All the goodness that that we talk about from Nutanix. 256 00:34:11.927 --> 00:34:15.089 Philip Sellers: Gyra, Harvey, thanks so much for spending some time with us. 257 00:34:15.860 --> 00:34:16.630 Jirah Cox: Like y'all. 258 00:34:17.359 --> 00:34:27.069 Philip Sellers: And for everybody listening. Hopefully, you'll join us back again very soon for another episode. And in the meantime we'll see you 259 00:34:27.655 --> 00:34:29.239 Philip Sellers: on that next episode.