Latest / Microsoft Cloud IT Pro Podcast / Episode 397 – Local LLMs: Why Every Microsoft 365 & Azure Pro Should Explore Them
Transcript
- 0:03Welcome to episode 397
- 0:06of the Microsoft Cloud IT Pro podcast recorded
- 0:09live on 03/10/2025.
- 0:12This is a show about Microsoft three sixty
- 0:14five and Azure from the perspective of IT
- 0:16pros and end users, where we discuss the
- 0:18topic or recent news and how it relates
- 0:20to you. We've been talking a lot about
- 0:23AI recently, particularly
- 0:24Microsoft Copilots.
- 0:26But what if you want to play around
- 0:28with AI outside of Copilot or chat GPT
- 0:31or any other hosted AI tool? In today's
- 0:34episode,
- 0:35Scott and Ben dive into the world of
- 0:37local LLMs,
- 0:39large language models, that run entirely
- 0:42on your device. We look at what models
- 0:44you can run, how you can integrate them
- 0:45into your workflow, and more.
- 0:49Oh, Scott. Here we are back in the
- 0:52stormy South. Stormy South. It has been stormy,
- 0:55but it's bright and sunny now. So I'll
- 0:57take it while I can get it. I
- 0:58don't have anything to go with Nordic. From
- 1:00the Nordic North, I'm back to the stormy
- 1:02South.
- 1:04From sea to shining sea and everything in
- 1:06between the seas? As long as you count
- 1:08Lake Michigan as the sea, which if you're
- 1:10from Michigan, you do. Like, East Coast, West
- 1:12Coast in Michigan are Lake Michigan and Lake
- 1:13Huron. We don't really count oceans in Michigan.
- 1:16Some of those lakes are kinda big, so,
- 1:18you might even say they're great. They could
- 1:20be great. It's always interesting.
- 1:22Side topic, coming down to Florida and talking
- 1:24to people about lakes and being from Michigan
- 1:26and Lake Michigan
- 1:27and Lake Superior and
- 1:29they're like, but it's a lake. And I'm
- 1:31like, yeah, but you can't see across it.
- 1:32So it kinda looks like an ocean when
- 1:34you're standing on the shore, and we get
- 1:36waves that are like, well, I think the
- 1:38biggest waves I've ever recorded in Lake Michigan
- 1:40were like 25, 20 six feet, and Lake
- 1:42Superior was up to 32
- 1:44foot waves. It's like these are not just
- 1:46like little lakes. These are massive bodies of
- 1:49water.
- 1:50They're really, really big ponds, you know.
- 1:53Joshua Foer: So now we're going across the
- 1:55pond and that means across Lake Michigan. Joshua
- 1:56Foer: An LLM what it thinks. Joshua Foer:
- 1:58We should ask an LLM because we're going
- 2:00to talk about LLMs. Joshua Foer: We're back
- 2:02to those things again. So I wanted to
- 2:04have a chat today
- 2:06about
- 2:07LLMs
- 2:07and
- 2:08running them locally. Like, I've been doing this
- 2:11more and more, and I think there's an
- 2:13interesting set of use cases and workflows. And
- 2:16I was having a chat with you,
- 2:17and this isn't something that you do in
- 2:20your kinda day today from what it sounds
- 2:22like, but maybe I can, like, get you
- 2:23in and and and hook you in a
- 2:25little bit along the way. Oh, you've already
- 2:27got me hooked. You sent me a few
- 2:28YouTube videos, and I started watching it, and
- 2:30the wheels started clicking.
- 2:31And I have one of the browser tabs
- 2:34up here. We'll put a link to it,
- 2:35Scott, about a use case that I already
- 2:37have for a local LLM.
- 2:39And you definitely got my wheels
- 2:41turning about
- 2:43what possibilities there are about how some of
- 2:45this works. In Microsoft three sixty five, I
- 2:48have played around with Copilot. I know a
- 2:50fair amount, but I've never really looked at
- 2:52running them locally and bewet my appetite
- 2:54for this. So this will be an interesting
- 2:57discussion, and I'm curious to see where it
- 3:00goes, your thoughts,
- 3:02my new
- 3:03thoughts. And my expanding list, Scott, you added
- 3:06something new to my list. I was doing
- 3:08so good. It's been a hot minute, but
- 3:10I I I think this is an important
- 3:12one. So as we talk about
- 3:14the
- 3:15kinda growth
- 3:17of
- 3:18generative AI
- 3:19and models along the way for,
- 3:23you know, certainly the the copilots of the
- 3:26world,
- 3:27the OpenAI's,
- 3:28Anthropic with Claude,
- 3:30DeepSeek with r one,
- 3:33all all these different kinds of things that
- 3:35exist out there. So
- 3:38they're they're nice that you can run them
- 3:39in a service.
- 3:40And I think most of us have kind
- 3:43of grown accustomed to that, and and it's
- 3:46it's a place that most of us are
- 3:47comfortable. Like, we know how to sign in
- 3:49to chat GPT on the web and maybe
- 3:51either
- 3:52have a chat with an LLM and and
- 3:56do some structured prompting and and try and
- 3:58get some responses out of it
- 4:00versus
- 4:01things like ChatGPT web search. And it's great.
- 4:03Right? It's it's all cloud based.
- 4:06Some of them are free. Some of them
- 4:07cost money.
- 4:09They really only start to get powerful
- 4:11when they do cost money. So now you're
- 4:14in the world where you're relying on this
- 4:16external service. You're gonna pay per request.
- 4:19And probably most importantly, there's a privacy angle
- 4:22here where you're sending your data out into
- 4:24the wild. Like, when you're chatting with Chat
- 4:27GPT in the web interface,
- 4:28you're passing that data to them. We saw
- 4:30this with DeepSeq. When DeepSeq kinda came out
- 4:33of the woodwork
- 4:34a couple weeks ago and the market freaked
- 4:35out. You know, they were about a month
- 4:37behind freaking out when it had actually been
- 4:38released. But that said, you know, DeepSeek immediately
- 4:42had a data leak and people broke in
- 4:44and they got all the usernames, they got
- 4:46the passwords, they got the prompts that were
- 4:48flowing through that system, things like that. So
- 4:50I think one of the most powerful things
- 4:52here is the ability to
- 4:55run a local LLM
- 4:57with
- 4:58data privacy in mind. So I'm gonna run
- 5:00these things locally. They're only going to be
- 5:02on my machine. They're not gonna communicate with
- 5:04the outside world. And then if you're in
- 5:06that world of, you know, being a little
- 5:09bit more cost conscious,
- 5:11you might wanna try some of these things
- 5:13out without paying per request in a service
- 5:15like chat, GPT, or Claude, or or something
- 5:18like that. And in that world, you're gonna
- 5:21also have a cost savings angle.
- 5:23You're gonna have offline capabilities.
- 5:25So the ability to chat with these models
- 5:28locally can be a little bit interesting
- 5:30and and how all that composes.
- 5:32And, you know, I think the kicker is
- 5:35most of us are geeks, and we run
- 5:37around with these really powerful computers. You know,
- 5:40you've got a laptop with gobs and gobs
- 5:42of RAM on it, and it's running a
- 5:45modern processor, it's got a GPU, it's got
- 5:48an MP,
- 5:49you know, you might be sitting there at
- 5:51home and you're like a PC gamer and
- 5:52that's how you,
- 5:54you know, just relax at the end of
- 5:56the day. Well, guess what? You got that
- 5:58monster GPU, you know, that fifty ninety or
- 6:00whatever that you can potentially use during the
- 6:02day with these things. And it turns out
- 6:04that you might actually chat with local LMs,
- 6:06like, more than you think. You know, like
- 6:08so we've talked about how we're Apple users,
- 6:11so iOS, things like that. The predictive text
- 6:14on iOS is all based on an LLM.
- 6:17It's based on a transformer.
- 6:19So that thing is running a local model.
- 6:21Well, you can run those similar models on
- 6:23your side. So it gives you this really
- 6:25interesting opportunity to kinda take advantage of AI
- 6:30while maintaining the privacy aspects, maybe letting you
- 6:32play with new things. Like, if you wanna
- 6:34play with DeepSeek without signing up for the
- 6:36DeepSeek service,
- 6:37like, hey, that that that that's a great
- 6:39way to do it. So we'll talk a
- 6:40little bit about that and kind of some
- 6:42of the advantages and what you can get
- 6:43on with. We should also talk about what
- 6:47folks can actually run,
- 6:49like, what's useful useful that can run locally
- 6:52for you. So we're gonna talk a little
- 6:53bit about, like, parameter size in a model
- 6:56and how big these things are. So turns
- 6:59out there's a big difference between a
- 7:021,000,000,000 parameter model, a 7,000,000,000 parameter model, a
- 7:0665,000,000,000
- 7:06parameter model, or, you know, like I said,
- 7:09if you wanna play around with DeepSeq,
- 7:10I was watching some videos on YouTube of
- 7:12people who are playing around with some clustered
- 7:15servers
- 7:16to do, like, 400,000,000,000
- 7:18parameter model runs. And, you know, you can't
- 7:20run, like, 400,000,000,000 parameters locally. You need, like,
- 7:22a distributed system, and, you you know, you
- 7:25can potentially do it across a series of
- 7:27servers within your premises. But that said, like,
- 7:30those aren't for everybody. They're gonna be too
- 7:32slow, cost a bunch for the GPUs, things
- 7:35like that. So we'll talk a little bit
- 7:36about that, about like parameters and, you know,
- 7:38maybe where more parameters doesn't always mean, like,
- 7:42better results. I think that's important too.
- 7:45There there's a little bit of nuance and
- 7:46kind of trade off here between
- 7:49speed of response, like how many tokens can
- 7:51an LLM respond back to you with, what's
- 7:53the accuracy of that, and probably most importantly,
- 7:56like what are the compute requirements on your
- 7:58end. So like the things that I'm gonna
- 8:00talk about that I run today, so I
- 8:01rock an m MaxBook Pro
- 8:04most of the time, and that's kind of
- 8:06like what I'm running on. And I've got,
- 8:07you know, 32 gigs of RAM in there,
- 8:09and and I'm all set in my world.
- 8:11You have a a different model on a
- 8:14different processor
- 8:15with more memory and potentially more GPUs, so
- 8:19you'll be able to run, like, maybe even,
- 8:20like, bigger things than I can run here.
- 8:22And that's okay. And then,
- 8:24you know, your mileage may vary. But it's
- 8:27kind of like anybody can get started with
- 8:28these things, even on, like, a little,
- 8:31you know, off the shelf NUC kind of
- 8:34PC or things like that. So beyond chatting
- 8:37with these things,
- 8:38you can also use them to empower your
- 8:40workflows.
- 8:41So you can use local AI models with
- 8:44Visual Studio Code. Like, you might sit out
- 8:46and go and say,
- 8:48I'm coding a dot net application.
- 8:51Let me go find the best LLM model
- 8:54for dot net applications,
- 8:56but I don't wanna pay for it. I
- 8:58I don't wanna, like, go to OpenAI or
- 9:00Anthropic and and do the cloud thing, anything
- 9:02like that. Well, maybe you can go out
- 9:03and actually just download a model and run
- 9:05it locally, and we'll kind of talk about
- 9:07the hosting engines for these things that expose
- 9:10things like standard OpenAI endpoints. So you can
- 9:12literally point Versus Code at a local LLM
- 9:15and have it write you PowerShell and all
- 9:17those things that are, like, private just to
- 9:19your machine
- 9:20without having to go out to the Internet
- 9:23and get those kinds of things done. So
- 9:25I think that's a fun little way to
- 9:26kind of think about integrating these things
- 9:29into your life and how they come together.
- 9:31So we just kind of want to go
- 9:33end to end and full circle between, can
- 9:35you run your own chat GPT
- 9:38like
- 9:39thing, model
- 9:41locally? And the answer is yes. So, yeah,
- 9:43like we should just kind of have a
- 9:45conversation
- 9:46about that. So
- 9:48why don't we start with like the whole
- 9:50data and privacy cost efficiency thing and all
- 9:53that stuff? I think that's one of the
- 9:54ones that can be super important
- 9:58that people think about. And kinda like you
- 10:00said, the deep sea click exposed to millions
- 10:02sensitive
- 10:03data records. One thing I've heard even when
- 10:05you start looking at things like ChatGPT
- 10:08versus
- 10:09Copilot and Microsoft three sixty five and going
- 10:11back to the local ones or doing OpenAI
- 10:14in Azure or
- 10:16something in AWS is it it very much
- 10:19goes back to where does that data go.
- 10:21Some people see rolling out Copilot as a
- 10:23security benefit because then they're not taking all
- 10:26that data from
- 10:27SharePoint, from Teams, from their Microsoft three sixty
- 10:30five tenant, sending it out into ChatGPT
- 10:33where it's escaping that Microsoft three sixty five
- 10:36boundary. OpenAI and Azure, same thing. If all
- 10:39your data's up in Azure somewhere, if you're
- 10:42working with Scott to store petabytes of data
- 10:45in blob storage and you want that to
- 10:47be used for OpenAI, you can do that.
- 10:49But then you do get into this local
- 10:50thing. All your data's
- 10:52local. Or one of the scenarios I have
- 10:54that we can put a link to is
- 10:56I use Home Assistant for all my smart
- 10:58home stuff because
- 11:00I like everything
- 11:01local. I don't want it all going out
- 11:03to relying on Samsung or
- 11:05any of those. What if you wanna integrate
- 11:08AI into your local smart home stuff and,
- 11:11again, you wanna keep it all internal? You're
- 11:14in an industry where
- 11:16you need to keep things on premises for
- 11:18some reason or certain regulations around that. I
- 11:21think there's a huge benefit to doing
- 11:23local AI, whether it's at that small
- 11:26in your house,
- 11:27you and I type scenario of
- 11:30smart home or something here or large enterprises
- 11:33that have very stringent data requirements and need
- 11:38to run it locally, maybe in their own
- 11:40data centers in clusters that they build internally
- 11:43and stuff. Home Assistant is a fun one.
- 11:45So if you think about AI and Home
- 11:48Assistant and what they're doing with like Home
- 11:49Assistant voice and some of those things, it
- 11:52relies on two paths. One is text to
- 11:55speech. So can I have Home Assistant talk
- 11:58to me? So some text goes in and
- 11:59can I have it talk back to me?
- 12:01And then it's also speech to text in
- 12:03the form of things maybe
- 12:05like Whisper,
- 12:07which is, you know, typically what I see
- 12:09integrated with most on that side. In fact,
- 12:11we use Whisper for generating transcripts sometimes for
- 12:15the show. So it's not just LLMs.
- 12:17It could be things like text to speech,
- 12:19speech to text. Could also be image generation.
- 12:22Like, if somebody is looking to, like, play
- 12:24around with stable diffusion, that that runs pretty
- 12:26well locally
- 12:28on most of these things as well. It
- 12:29could be a little bit slow, but, hey,
- 12:31that that's okay. That's that's part of the
- 12:33trade off of not having to pay and
- 12:35and push these things through. But I I
- 12:38think the most important thing is just when
- 12:39you're running an LLM locally, you're basically mitigating
- 12:44a bunch of that risk
- 12:46of having to worry about compliance, having to
- 12:48worry about legal concerns. Like, hey, I'm submitting,
- 12:51like, this thing that's important to me. Like,
- 12:54I'm never like, for example, I'm never going
- 12:56to chat with my taxes
- 12:58with anything other than, like, a local LLM
- 13:00to help me break some of that stuff
- 13:01down.
- 13:03But, you know, somebody else might be out
- 13:05there, but good good luck when you're when
- 13:07you're in the next data breach or or
- 13:08or whatever happens. So there's things like that.
- 13:11I think the other one that's important to
- 13:12consider is kind of the cost angle of
- 13:15things. Like, I'll be the first to admit
- 13:17that I'm pretty frugal. So if you're thinking
- 13:20about maybe like OpenAI and having to go
- 13:22out and pay for OpenAI, and you're either
- 13:24paying per request or you're on one of
- 13:27the monthly plans. And those can get pretty
- 13:28expensive. Right? If you wanna get up there,
- 13:30you can spend up to, like, $200 a
- 13:31month. But typically, they're on the order of,
- 13:34like,
- 13:35you know, 1¢ US per 1,000 tokens.
- 13:39And then you're like, Well, what's a token?
- 13:40Like, how many words comprise a token? Like,
- 13:43it can be a little bit weird to
- 13:44figure out the pricing. So sometimes you just
- 13:45want to play around with these things locally
- 13:48without having that cost constraint,
- 13:51because costs can run away from you pretty
- 13:53quickly, especially if you're being like super chatty
- 13:57and doing longer chat threads and things like
- 14:00that. Or the other place they tend to
- 14:02get pretty expensive
- 14:03is if you're integrating
- 14:06these
- 14:07AIs
- 14:08into,
- 14:09like, your coding workflows, like, hey, you're you're
- 14:11out there and you're sitting there and you're
- 14:12like, I want a vibe code. Well, great.
- 14:15When you're like vibe coding across 10,000 lines
- 14:17of code, it starts
- 14:19to add up and get pretty expensive. So
- 14:21you already bought this, you know, honking computer.
- 14:24You got a GPU. It's got CPU.
- 14:26It's got a fast disk. You might as
- 14:28well use it for a little bit more
- 14:31than just writing your PowerShell scripts. Like, why
- 14:33why are you sitting there writing in Versus
- 14:35Code by hand when, you know, you could
- 14:37be just vibing your way through that stuff?
- 14:39For sure. And I think that's one thing.
- 14:41I guess I kind of always realized it
- 14:42in the back of my head, comparing local
- 14:44LLM
- 14:45to JetGPT to Copilot to cloud based, it
- 14:48kinda struck me that
- 14:50from a pricing perspective, when you're using cloud
- 14:53based LLMs, you're not paying for the models.
- 14:55Like, these companies, these models are all out
- 14:58there,
- 14:59whether it's
- 15:00DeepSeek
- 15:01or Lama
- 15:03or any of those. What you're really paying
- 15:05for is the compute to
- 15:07process the request to these models, and that's
- 15:09where that cost comes in. Do you wanna
- 15:11spend it in on premises hardware and hardware
- 15:13running in your house, or are you giving
- 15:15it to these cloud providers for the hardware
- 15:18out there running models that maybe you don't
- 15:21physically have the capability of running on your
- 15:24compute that you own? It is an interesting
- 15:26one. The other thing that, you know, like,
- 15:28once you get a little bit more advanced
- 15:29and you start going down the path of
- 15:30some of this stuff, if you really get
- 15:32into it, you start looking at things like
- 15:35fine tuning
- 15:36and doing RAG or retrieval augmented generation against
- 15:39things. So we'll put a link in the
- 15:41show notes to a Network Chuck episode where
- 15:44he talks about running local LMs. And one
- 15:46of the things that he does, he has
- 15:47this really interesting use case where when he
- 15:50attends church, all the sermons are transcribed,
- 15:53and he uses local LLMs to summarize the
- 15:56sermons for himself. Like, he doesn't always get
- 15:58to attend live, but he still wants to
- 16:00get the messaging out of it. So he
- 16:02does all that stuff like local LLM, and
- 16:05it's just all there ready to go. It
- 16:07does the transcription,
- 16:08like pulls it all off a YouTube thing,
- 16:10transcribes it, runs it through an LLM, gives
- 16:12him the summary, and then that summary is
- 16:14written back as a markdown file where it
- 16:17lands in Obsidian,
- 16:18and then he can just use his network
- 16:21brain in Obsidian to go and figure some
- 16:23of that stuff out too. So you can
- 16:25get pretty rich with these things if you
- 16:27start to kinda, run through the use cases.
- 16:29So we're, like, Network Chuck might be doing
- 16:31I might be working on coding a new
- 16:34application,
- 16:35and I just want it to learn off
- 16:37maybe an existing code base from, like, the
- 16:39previous two versions or iterations
- 16:41or things like that that I did along
- 16:43the way. So you can also do these
- 16:44things like fine tuning and get up and
- 16:46running
- 16:47pretty pretty quickly. It's actually, like, turns out
- 16:50a lot of the tooling's already out there.
- 16:51Like, these things are
- 16:53not the hardest thing to stand up. But
- 16:56before we stand them up, we should also
- 16:57probably talk a little bit about, like, what
- 16:59kinds of models you can run
- 17:02because your mileage may vary here based on
- 17:05your your hardware and what's available to you,
- 17:08your your network bandwidths, and a couple other
- 17:11things.
- 17:15Do you feel overwhelmed by trying to manage
- 17:17your Office three sixty five environment? Are you
- 17:19facing unexpected issues that disrupt your company's productivity?
- 17:23Intelligink is here to help. Much like you
- 17:25take your car to the mechanic that has
- 17:27specialized knowledge on how to best keep your
- 17:29car running, Intelligink helps you with your Microsoft
- 17:32cloud environment because that's their expertise.
- 17:35Intelligink keeps up with the latest updates in
- 17:37the Microsoft cloud to help keep your business
- 17:39running smoothly and ahead of the curve. Whether
- 17:41you are a small organization with just a
- 17:43few users up to an organization of several
- 17:46thousand employees,
- 17:47they want to partner with you to implement
- 17:49and administer your Microsoft cloud technology.
- 17:53Visit them at inteliginc.com/podcast.
- 17:56That's intelligink.com/podcast
- 18:04for more information or to schedule a thirty
- 18:06minute call to get started with them today.
- 18:08Remember, Intelligink focuses on the Microsoft cloud so
- 18:11you can focus on your business.
- 18:15So talking hardware, do you wanna drive into
- 18:17hardware or models? Where should we go? It's
- 18:20kinda like a both conversation.
- 18:22So I think we can cover kind of
- 18:23the whole parameterization
- 18:25question
- 18:26and how big these things are to run
- 18:29locally
- 18:29along with some of the hardware
- 18:31constraints
- 18:32that
- 18:33come along with them. So when you think
- 18:35about the models that you can run, one
- 18:37of the first things that's gonna happen is
- 18:39you might go out and grab Ollama,
- 18:42you might grab LM Studio. You're you're gonna
- 18:45grab some system that's going to let you
- 18:47basically
- 18:48run that model and be able to run
- 18:51prompts against it. So though those models are
- 18:54gonna have different
- 18:56sizes,
- 18:57and those sizes equate back to parameters. So
- 19:00you're gonna go out and you're gonna see
- 19:01things like, oh, I wanna run llama three.
- 19:04And llama three might have,
- 19:07you know, a 7,000,000,000 parameter model. It might
- 19:10have a 300,000,000,000
- 19:12parameter model. It could have a 1,000,000,000 parameter.
- 19:15It could have something that's even smaller than
- 19:16that. So these things start to kind of
- 19:19become important. So if you're thinking about, like,
- 19:21parameters, number of parameters in a model, which
- 19:23is going to equate to
- 19:25kind of functionality
- 19:27within that model,
- 19:28In some place like a 7,000,000,000 parameter model,
- 19:31if you're looking at, like, LAMA two seven
- 19:33b, you're looking at Mistral seven b, like,
- 19:36those are pretty good starting points, and you
- 19:38don't need a super monster laptop or desktop
- 19:41to do it, just something decent. So if
- 19:43you have about 16 gigs of RAM and
- 19:45some CPU, you're good. Like, you don't need
- 19:47a dedicated GPU. You can absolutely do this
- 19:50stuff on CPU.
- 19:51I hesitate to say fast. It'll be fast
- 19:54ish. It might feel a little bit slow,
- 19:56like you'll see, like, the words typing out
- 19:58on screen, but that's okay. That that kind
- 20:00of equates to the experience that you might
- 20:01have in a chat GPT or or a
- 20:03Claude or things like that.
- 20:05But they're also super lightweight.
- 20:07So you you can get models that potentially
- 20:10when you download the model, they're measured in,
- 20:12like, hundreds of bags.
- 20:14Some are in the gigabyte range. Like, if
- 20:15you're in, like, a 7,000,000,000
- 20:17parameter model, you're talking about maybe, like, two
- 20:19to three gigs of downloading a quantized model
- 20:22and being able to track against it. And
- 20:25with 7,000,000,000 parameters, you'll probably find
- 20:28that they're good enough for most tasks,
- 20:32for most
- 20:33personal tasks. Hey. Summarize this for me. Hey.
- 20:36Give me a quick idea of this.
- 20:38Translate this to this. Like, those kinds of
- 20:41things, it's perfect. Hey. I wanna pump in
- 20:43the transcript from a YouTube video and have
- 20:46a local model summarize it for me. That's
- 20:48an awesome job for, like, a 3,000,000,007
- 20:50parameter model,
- 20:52things like that. You can get a little
- 20:54bit bigger, and a little bit bigger is
- 20:56typically gonna be in the something of, like,
- 20:5810 to 30,000,000,000
- 21:00parameter range.
- 21:01So
- 21:02now you're getting a little bit more honking.
- 21:04You're actually gonna need some GPU here, and
- 21:07you're probably gonna need more RAM as well.
- 21:10So, like, 16 gigs of RAM isn't gonna
- 21:11cut it. You're probably gonna need something closer
- 21:13to 32 gigs of RAM.
- 21:16You're gonna need some kind of GPU to
- 21:18drive that.
- 21:20You know, I think you could maybe get
- 21:21by on, like, an RTX thirty ninety or
- 21:23something like that. You'd probably wanna be in,
- 21:25like, a a a 40 series, like, a
- 21:27forty sixty, 40 70. Or if you're all
- 21:29on board and, like I said, you're a
- 21:31PC gamer and you've got that fifty ninety
- 21:33sitting in there, like, go ahead. Use it.
- 21:35It's ready to go. Nobody has the 50
- 21:37series. There were only, like, 10 of them
- 21:39produced and nobody could buy them. Well, and
- 21:40out of the 10 that were produced, 10
- 21:41out of 10 were broken, so the the
- 21:43yields are great. And melted power cables. Okay.
- 21:46Anyways, sidetracked. Yes. But you're gonna need one
- 21:48of those high end GPUs. Yeah. Well, you're
- 21:49gonna need a GPU. Like, I think the
- 21:51difference between, like, a 3,000,000,000, seven parameter model
- 21:54and then you get up to those, like,
- 21:5510 to 30 range
- 21:57is, do I need a GPU or do
- 21:59I not need a GPU?
- 22:00So you can do the smaller models just
- 22:02with CPU as long as you have enough
- 22:04RAM. At some point, you're gonna want GPU
- 22:06as well to go ahead and offload those.
- 22:10So if you're thinking like, hey, my use
- 22:12case for running a local LM is doing
- 22:15advanced coding, like, I'm I'm beyond, like, summarization,
- 22:17and I want this thing to help me
- 22:19write applications,
- 22:20PowerShell scripts, bash scripts, anything like that, you're
- 22:24probably gonna wanna be in that range where
- 22:26you've got a little bit more RAM and
- 22:28you've got a GPU,
- 22:29and then you kinda find the model that
- 22:31you like, and and that ends up being
- 22:33your sweet spot there. After that, you get
- 22:35into, like, the big, big models. So you're
- 22:37into, like, 65. I think, I was watching
- 22:40another NetworkChuck video. He ran one on a
- 22:42cluster
- 22:42of Those studios. I think it was the
- 22:44m one studios. It was, like, a cluster
- 22:46of, like, six of those where he was
- 22:48able to run, like, a 400,000,000,000 parameter model,
- 22:50but it was only able to output context
- 22:53you know, like one
- 22:55word,
- 22:55a second. Like, it's just so slow that
- 22:58it's that it's not actually useful. Right. So
- 23:01slow. A few times it looked like it
- 23:02even got stuck and,
- 23:05yeah, it was it was interesting. We'll put
- 23:07a link to that video in the show
- 23:08notes too. Yeah. So the way I think
- 23:10about that, the really big models, they're basically
- 23:13not there for, like, the faint of heart.
- 23:16They're there if you know what you're doing,
- 23:17if you've got the hardware to back it,
- 23:19both CPU,
- 23:20RAM,
- 23:22and and GPU.
- 23:23So if you think about it, like, there's
- 23:25kinda like a way that you can just
- 23:26break it down into a simple set of,
- 23:27like, pros and cons. So when you're sitting
- 23:29out there, you're in that, like, three, five,
- 23:32seven billion range,
- 23:34that's gonna be fast. You can do it
- 23:36on simple low hardware,
- 23:37or you can even do it on beefier
- 23:39hardware. Like in my case, like when I'm
- 23:41on my M1 Max, typically, I'm also running
- 23:43Windows in a virtual machine. So that's typically
- 23:46got half my RAM already. And then I've
- 23:48got a little bit of RAM that's going
- 23:49to the OS and things like that as
- 23:50well. So even if I could run a
- 23:52bigger model, I'm not going to because I'm
- 23:55still having resource contention and other things. Like,
- 23:57sometimes I don't wanna shut down my VM
- 23:59or I don't wanna shut down Versus Code
- 24:00because I'm I'm using those things. Right. You
- 24:02know, smaller models, fast,
- 24:04commodity hardware,
- 24:06good enough for
- 24:07easy tasks. Like, sum summarize that transcript for
- 24:11me thing, they're gonna be great for that.
- 24:13You get into that middle range, probably your
- 24:15sweet spot, like, if you do have a
- 24:16little GPU to drive these things,
- 24:19good accuracy,
- 24:20more context awareness,
- 24:22and kinda longer context windows. So as you're
- 24:25chatting with these things, they can remember,
- 24:27quote, unquote, big air quotes here. They can
- 24:29remember what you previously typed with them. So
- 24:32having bigger context windows and and more RAM
- 24:34and VRAM from your GPUs to host those
- 24:36context windows in becomes a little bit important.
- 24:39And then, like, if you're,
- 24:41you know, a monster gamer, you've got just
- 24:43a bunch of these things laying around and
- 24:45you wanna network them all together, it's super
- 24:47easy to do that too if you got
- 24:48enough hardware running around,
- 24:50and and you can go and,
- 24:52make that happen. So once you've kinda figured
- 24:55out your your hardware and you've got a
- 24:57sense for what you wanna do and what
- 24:59you're gonna be able to run locally, well,
- 25:01then you need a way to
- 25:03run these
- 25:04things
- 25:05locally, which, you know, it's not a little
- 25:08decision to make. Right. And another thing about
- 25:10the hardware that I found interesting watching the
- 25:12NetworkChuck videos as well was because we talked
- 25:15about the Macs, Macs have, like, that shared
- 25:17memory. They don't have dedicated video memory and
- 25:21system memory. So one thing he was doing
- 25:23was when he was running these models, like,
- 25:25all the memory was going to process the
- 25:28model because it doesn't have, like, those physical
- 25:31boundaries
- 25:32between
- 25:33physical and system memory. So I think that
- 25:35was another thing to watch out for. And
- 25:36the other thing, because you mentioned networking,
- 25:38he also found, like, running a 10 gig
- 25:40network. Something I didn't realize because I've never
- 25:43done this locally, how chatty these are if
- 25:45you're running a cluster over a network. Super
- 25:47chatty. He'd, like, saturated his 10 gig network,
- 25:51and that appeared I would say, I don't
- 25:53know that it was definitive in his videos,
- 25:55but appeared to be the bottleneck
- 25:56using these,
- 25:58clustered studios.
- 25:59So then he switched to Thunderbolt, which gave
- 26:02him a 40 gig network essentially. And even
- 26:05that, he managed to saturate, get a little
- 26:07bit more speed out of it using Thunderbolt
- 26:09as opposed to a 10 gig network. But
- 26:12if you do start thinking of clustering
- 26:14larger models,
- 26:16networking is also huge when it comes into
- 26:19the hardware for these things. I don't really
- 26:20get into the network model kind of thing.
- 26:23Like, I just don't have enough hardware running
- 26:25around here at home to do it. I
- 26:26I certainly think it's interesting if you can
- 26:29get there. So, yeah, we can kinda talk
- 26:31about that maybe with, like, more advanced stuff.
- 26:33Yes. So on to software. So you got
- 26:35your hardware. You got your software. I keep
- 26:38seeing you sent LM Studio. I've not looked
- 26:39at LM Studio. The one that always seems
- 26:41to pop up for me both in
- 26:44the Home Assistant as well as in a
- 26:45lot of the network check is Ollama for
- 26:47running these locally. Your decision here is
- 26:51how geeky do you wanna be and and
- 26:53what is your workflow?
- 26:54So if your primary workflow
- 26:57is
- 26:58you just want to
- 26:59chat with a a chatbot, like, you wanna
- 27:02hop right in, you wanna download a model,
- 27:03and you wanna be able to chat right
- 27:05away in, like, a nice GUI and a
- 27:07graphical interface,
- 27:08LM Studio is great for that. There's like,
- 27:11if you go out and you look this
- 27:12stuff up and you hop on Reddit or
- 27:14things like that,
- 27:15there there's going to be that set of
- 27:17folks out there who hate LM Studio
- 27:20because it's closed source,
- 27:22but, you know, I I'm just looking to
- 27:24play with these things. So for what I
- 27:26wanna do, it certainly
- 27:28works
- 27:29works great. Comes together, does what I need
- 27:31it to do. That said, you can also
- 27:33do Ollama,
- 27:35and Ollama is gonna be more command line
- 27:38driven, like you're gonna do more installations from
- 27:40the command line, you're even gonna download your
- 27:42models from the command line, so you're kinda
- 27:44trading off ease of use there. There's pros
- 27:46and cons to both depending on what you're
- 27:48doing. LM Studio is great if you just
- 27:50want to chat, you want to immediately have
- 27:52OpenAI spec ed endpoints
- 27:54exposed maybe to things like Versus Code locally,
- 27:57and you just don't want to wire anything
- 27:58up. You're looking for just like a one
- 27:59shot install, and you're going to be one
- 28:01and done. The other way you can do
- 28:03it is you can go to Ollama, and
- 28:05you can find your model that you wanna
- 28:07run on there. So, you know, I wanna
- 28:09run llama two seven billion, and you'll go
- 28:12download that, and you're gonna do all this
- 28:14from the command line. Now you wanna chat
- 28:16with that thing.
- 28:18Well,
- 28:19you can certainly chat with it from the
- 28:20command line. That that's totally a possibility. If
- 28:23if that's your jam or your jelly, awesome.
- 28:25Go for it. But if you want to
- 28:27chat with it in a GUI, now you
- 28:29gotta go install something else. Like, you might
- 28:31have to go install
- 28:33open web UI
- 28:34to to to get that piece going and
- 28:36and stand all that up. So it's not
- 28:38like it's hard to do. It's just your
- 28:41your flavor and and and where you sit
- 28:43and where you wanna land.
- 28:44You know, if I'm looking to just do
- 28:46things quickly and, like, I'm just in there
- 28:48to maybe, like, oh, hey. I see Microsoft
- 28:50released a new model for 05/04,
- 28:53and they they were they, you know, just
- 28:55pushed new models for 05/03 and '5 '4,
- 28:57and I I wanna compare those two things.
- 29:00I'll probably just spin those up in LM
- 29:02Studio. Super easy. Next, next, next my way
- 29:04through it. I don't have to remember a
- 29:06bunch of command line parameters, things like that.
- 29:08If I'm doing more like application development and
- 29:11I'm thinking about, like, hey. I want to
- 29:13stand this thing up. I wanna have it
- 29:14running in the background. I want some endpoints
- 29:16that are exposed. Maybe I can build, like,
- 29:17an app that's doing, like, some light rag
- 29:19or some fine tuning on top of it,
- 29:21and I've got, like, a Python script over
- 29:23here that needs to talk to the model.
- 29:25Awesome. Great. Like, that's that's where Ollama sits,
- 29:28and it has its space ready to go
- 29:31for you. So much like picking a model
- 29:34size, you're you're just doing a pros and
- 29:36cons and a little bit of a trade
- 29:37off thing. So Ollama, if you want a
- 29:39simple command line experience and you're comfortable at
- 29:42the terminal, go for it. Windows, macOS, Linux,
- 29:45it's all there. LM Studio, if you're not
- 29:47opposed to closed source and you just want
- 29:51a GUI from the start for all the
- 29:52things, for downloading, for chatting, for,
- 29:56all all that stuff. Again, macOS, Windows, Linux,
- 29:59ready to go. It's just closed source versus
- 30:01open source is really how I think about
- 30:03it. And then if you really do go
- 30:05down the Ollama path, you're probably gonna end
- 30:08up in a space where you wanna run
- 30:09a local chat UI, like a web based
- 30:12chatbot style thing,
- 30:14and
- 30:15then you'll just use something like Open Web
- 30:17UI for that. And, again, super easy to
- 30:19install. You're just basically hosting a little
- 30:22a little web server locally that knows how
- 30:25to
- 30:26chat with chat with that model. And then
- 30:29it could be a little bit different depending
- 30:31on like the extension tooling that you're going
- 30:32to use from there. So I talked about
- 30:34maybe like integrating Versus Code with one of
- 30:36these locally.
- 30:37So if you're doing
- 30:39Versus Code, you're gonna typically go grab an
- 30:41extension. So there's things like CodeGPT,
- 30:44there's continue dot dev,
- 30:46there's an Ollama extension, which can actually just
- 30:48talk natively to your Ollama endpoint.
- 30:51Or like I said, LM Studio exposes OpenAI
- 30:55compatible
- 30:56endpoints. So that's kind of a known, like,
- 30:58you know, web interface that you can throw
- 31:00a request at in a structured way,
- 31:02and it will respond in a in a
- 31:04way that most of the extensions
- 31:07are going to understand
- 31:09and get you ramped up for and ready
- 31:11to go with. Yeah, looking through this and
- 31:13most of the videos I saw, and again,
- 31:14were all Olamae, even the command line based
- 31:17looked really
- 31:19simple, lots of guides to just walk through,
- 31:22type this in, this is how you tie
- 31:24that in, this is how you go stand
- 31:25up the WebUI,
- 31:27point WebUI, to all of those.
- 31:30So none of this
- 31:31really seemed that complicated in everything I watched
- 31:34and, again, made me excited, like, I need
- 31:36to go try this out and go find
- 31:38a computer that I can
- 31:40absolutely bury with a model. See what I
- 31:43can do. See what damage I can do
- 31:44to my computer, Scott. It is not hard
- 31:46to do. So the other thing that you
- 31:48can do, if you're comfortable on the command
- 31:50line, there's another project out there that's called
- 31:52Fabric.
- 31:53So Fabric is kind of a
- 31:56it it allows you to easily network and
- 31:59distribute traffic across multiple nodes, but you can
- 32:02also do it on a single node. So
- 32:03So I was talking earlier about, like, that,
- 32:05you know, sermon summarization thing. Yep. And that's
- 32:07all based on Fabric. So Fabric, again, command
- 32:10line, it can run with local LLMs. It's
- 32:12a little kinda
- 32:14opaque for for how it does it. So,
- 32:15you know, make sure you download one of
- 32:17the the newer versions of it, And Fabric
- 32:19is all run from the command line as
- 32:21well. But then you can super easily integrate
- 32:24Fabric into things like bash scripts. So, like,
- 32:27I use it for the same thing. Like,
- 32:29if I think about the the podcast, I
- 32:31just have a bash script that runs Whispir
- 32:33locally. So Whispir is
- 32:36a speech to text model Yep. That OpenAI,
- 32:38and I can run that locally. Like, that
- 32:39runs on my hardware just fine. So I've
- 32:41just got a little bash script that takes
- 32:43that, generates the transcript, and then I just
- 32:45pipe the summaries out into Fabric to have
- 32:48those for myself
- 32:49in just my notes on the side. Right?
- 32:51Like, hey, here's the things we talked about
- 32:53and and how they're coming together. So
- 32:56very, very, very easy to get on with
- 32:58this stuff. And I think for most of
- 33:00our audience as well, like you folks are
- 33:01all comfortable on the command line. You don't
- 33:03need a GUI for this stuff. You can
- 33:04follow some instructions
- 33:06and wire these up. And we're not talking
- 33:08like super complicated things. We're basically talking the
- 33:10equivalent of like a brew or a chocolatey
- 33:13install or a Winget install, like just little
- 33:15one liners to get all this stuff up
- 33:17and running. Absolutely. You don't need to go
- 33:18write 50 line PowerShell scripts or pipe a
- 33:21bunch of things. It's really straightforward from everything
- 33:24I saw. Super easy to get up and
- 33:26going with that. I would say, like, the
- 33:28other thing you might wanna do a little
- 33:30bit is
- 33:31when you're exploring models.
- 33:33So if you go into, like, LM Studio
- 33:35and you're going through their model catalog or
- 33:37you're on, Ollama and you're exploring their model
- 33:40catalog, you might wanna just start with, like,
- 33:42some of the more popular ones to get
- 33:45up and running. So, you know, there there
- 33:47are differences between these things,
- 33:50you know, depending on what you're doing. Like,
- 33:52you can't go ask DeepSeek
- 33:54what happened in Tiananmen Square. Like, that is
- 33:56not programmed into that model,
- 33:58e even in the one that you you
- 34:00download and
- 34:01you run locally, but, you know, you can
- 34:03do that with, other stuff. So these models
- 34:06all vary. The other thing that you can
- 34:07do is you can go through the model
- 34:09catalogs,
- 34:09and you can find models that are purpose
- 34:12built for certain things.
- 34:14So there are models that are generated within
- 34:16these families. So you talk about, like, LAMA.
- 34:19There's gonna be versions of the LAMA model
- 34:21that are better for doing coding assistance things
- 34:23with it than there are for doing just
- 34:26straight one shot text summarization,
- 34:29stuff like that. So
- 34:30you you have to think through that a
- 34:32little bit too, like, just what's your workflow
- 34:35and
- 34:36what are you trying to
- 34:38get at
- 34:40along the way?
- 34:41And then be prepared for a little bit
- 34:42of latency
- 34:44and maybe differences in perf when you're running
- 34:46with these things. I think lots of people
- 34:48set out and they say, oh, I'm gonna
- 34:49be able to run that model locally, and
- 34:50it's gonna be so much faster because it
- 34:52doesn't need to go out and talk to
- 34:53the Internet. Like, it doesn't need to talk
- 34:55to Claude. It it it doesn't need to
- 34:57talk to chat GPT, anything like that. Yeah.
- 35:00Like, absolutely. You've eliminated the latency of that
- 35:03whole, like, request response thing having to traverse
- 35:05the Internet,
- 35:06but you still have to have the hardware
- 35:08that's capable of running this and standing it
- 35:10all up. So you might wanna even, like,
- 35:12play around before you integrate these things. Like,
- 35:13if you're interested in, like, a coding workflow
- 35:16with or integrating with Versus Code, things like
- 35:18that, you'll probably wanna play around with the
- 35:20the models a little bit locally to find
- 35:21the one that's got the the sweet spot
- 35:23for you based on
- 35:25number of parameters, your hardware, things like that
- 35:27before you go down the path of integrating
- 35:29it in Versus Code and then being disappointed
- 35:30that it's too slow or or things like
- 35:32that. There's a lot of blogs out there
- 35:34that'll just tell you, like, oh, running AI
- 35:37locally, like, it's super fast. It's it's super
- 35:39easy. It is super easy. It's not always
- 35:41super fast. So you so you do have
- 35:43to be prepared for that depending on your
- 35:44hardware. Yeah. Along with the model, Scott, this
- 35:47is another thing again, being fairly new to
- 35:49this, have you
- 35:50compared at all? Because another thing you can
- 35:52run into is quantization of these models. Right?
- 35:55And this is something else Network Chuck talked
- 35:57about in one of his where some of
- 35:59these larger models, they quantize.
- 36:02I don't know if that's the word. They
- 36:03quantize them down, and it sounds like it's
- 36:06essentially taking
- 36:08different aspects of the model. And inside
- 36:10models, they have model weights with, like, 32
- 36:14bit precision, and they reduce these down to
- 36:16eight bit, four bit precision,
- 36:18which makes them not as accurate but makes
- 36:20them smaller so you can run a
- 36:24larger model. Some of those bigger ones we
- 36:26talked about like 65,000,000,000
- 36:28plus parameters
- 36:30on less hardware,
- 36:31but with more
- 36:34not the accuracy
- 36:35versus running maybe a model with less parameters,
- 36:38but you get the full
- 36:40the full model weights in there where you're
- 36:42running the 32 bit precision instead of quantasize
- 36:45them down. Again, when you're downloading models, definitely
- 36:47something to watch out for because if these
- 36:50are quantasized
- 36:52and they have smaller, they can be less
- 36:54accurate, you can run them. Like, have you
- 36:56ever compared those of let's go run a
- 36:5830,000,000,000 parameter model on local hardware versus a,
- 37:0365,000,000,000
- 37:05model or parameter model that's quantized down to
- 37:08eight bit instead of 32 bit? I don't
- 37:10think many folks are running 32 bit. Most
- 37:13are probably running
- 37:15Four bit. Some kind of like well, something
- 37:17like 16 or lower, so like four, eight,
- 37:2116. I think when you go out and,
- 37:22like, you watch a lot of YouTube videos
- 37:24and and,
- 37:25you know, if if you do go down
- 37:27this path and you start getting into it,
- 37:28I think YouTube is a great place to
- 37:29go to and start to see. You'll see
- 37:31lots of people playing around with massive models,
- 37:35but with a
- 37:37like, only, like, four bits. Right. So they're
- 37:39doing that just so they can run it,
- 37:40not so they can run it effectively to
- 37:42drive a workflow. Like, they're just trying to
- 37:44try it out and see how many tokens
- 37:46a second they can get out of it
- 37:47or something like that. So a four bit
- 37:49model is
- 37:51absolutely going to
- 37:53run on, like, consumer grade GPUs, CPUs.
- 37:56Like, you're gonna be all good, ready to
- 37:58go there, but you have to know that
- 38:00it's been extremely compressed. So it can get
- 38:03it down to a smaller download size,
- 38:05and thus, it's going to take less memory
- 38:08and less processing power to go ahead and
- 38:10run it. So you might be running like,
- 38:13you know, like if I think about, like,
- 38:14the transformer that's running in iOS,
- 38:17that's probably
- 38:19a a a four bit model. Right? Like,
- 38:20it's sitting there. It's running on commodity hardware
- 38:22Right. And it's just doing what it needs
- 38:24to do. Now, if I'm on my desktop
- 38:27or or my m one
- 38:29MacBook, you know, I might be thinking about
- 38:31an eight bit model, and I'm okay with
- 38:33the performance trade off. Like, I'm I'm okay
- 38:36if it chats with me at, like, you
- 38:37know, like, two tokens a second kinda thing.
- 38:40Like, it can be super slow. It's it's
- 38:42okay. But you're not gonna run these, like,
- 38:44massive models because those are absolutely running in
- 38:48those
- 38:49massive data centers and and that set of
- 38:51infrastructure. Like, I I just wanna be clear.
- 38:54Like, you can't do the things that, like,
- 38:56ChatGPT
- 38:57can do with, like, o one running in
- 38:58their data center
- 38:59locally at your house. Like, that's just not
- 39:01the way these things work. It's it's not
- 39:03how they come together. So if you think
- 39:05about, like, the the whole quantization thing, it's
- 39:07all about
- 39:08packing things down and basically, like, archiving them,
- 39:11right? Put a tar or zip together of
- 39:13this
- 39:14thing and reduce the size, reduce the computational
- 39:17requirements,
- 39:18all that kind of stuff. So you're going
- 39:21to get small models. Hey, that's great. They're
- 39:23going to use less memory,
- 39:24and you might be able to run a
- 39:26larger model. Like, you could run a four
- 39:28bit, you know, 30,000,000,000 parameter model, but it's
- 39:31gonna be less accurate. And is accuracy important
- 39:34to you? Well, you might wanna go to,
- 39:36like, an eight bit like, 7,000,000,000 parameter model,
- 39:39something like that. So it it's gonna be
- 39:41very dependent on, like, your workflow
- 39:43and
- 39:44your
- 39:45use case for these things.
- 39:47I think the biggest thing you miss out
- 39:49on is accuracy.
- 39:51So, you know, like, if I'm summarizing
- 39:53the podcast transcripts, I want those to be
- 39:55kind of accurate. Like, I I don't want
- 39:57them to just be hallucinating all over the
- 39:59place.
- 40:00But,
- 40:01you know, if I'm doing something else, like,
- 40:04hey. Help me write a poem about, you
- 40:06know, iPads. Like, whatever. Do it with all
- 40:09the least accuracy
- 40:10that you want out there
- 40:12along the way. I think the most common
- 40:14thing, like so the other thing you run
- 40:16into with quantization
- 40:17is
- 40:18there there's a bunch of different methods for
- 40:20this. So
- 40:22there's
- 40:23Q,
- 40:23which is basically like four bit quantization.
- 40:27There's another format called
- 40:29g g u f. So that's kind of,
- 40:32like, the standard for running these things
- 40:35efficiently. So you'll see a lot of these
- 40:36things when you go in like, what's the
- 40:38format of the model? Oh, it's a g
- 40:40g u f. I don't even know how
- 40:41it's pronounced.
- 40:42But, you know, you can go in and
- 40:44and grab those things
- 40:46and and figure those out. So you can
- 40:48think of, like, quantization maybe as, like, another
- 40:50weight that you can put on that scale
- 40:52when you're trying to find that balance between
- 40:55model size, parameter count, quantization,
- 40:58and the hardware that you run and the
- 41:00workload that you wanna do. So how does
- 41:02that scale tip and where do you wanna
- 41:04land? It just becomes an another consideration in
- 41:06there for you. Sounds good. Anything else before
- 41:09wrapping this episode up? So a couple things.
- 41:12If folks haven't done this yet, like Go
- 41:13do it. You should totally go out and
- 41:16just try and play around with Ollama LM
- 41:18Studio.
- 41:19If you're already doing it today, come back
- 41:20and give us some feedback. Let let us
- 41:22know what you're using it for. I think
- 41:24there's all sorts of interesting use cases for
- 41:27this stuff. We're just getting Ben started on
- 41:29his list. Let's make his list a lot
- 41:31longer for things that he is missing out
- 41:34in his life. Home assistant and AI. He
- 41:36needs to do to run a
- 41:38chat model locally. And then if you're doing
- 41:40other things besides chat models, like I said,
- 41:42there's the stable diffusions of the world, there's
- 41:45image generation, there's whisper, there's all these other
- 41:48things out there. I was very surprised at
- 41:50how approachable they are. I always thought this
- 41:52was going to be like mystical dark arts
- 41:55and magic and not for mere mortals kind
- 41:57of thing. It's very much for mere mortals,
- 41:59Like, super easy to get started with, super
- 42:01turnkey,
- 42:02and I would guarantee that almost anybody who
- 42:04listens to this podcast probably has the hardware
- 42:06to run this stuff and make it happen.
- 42:08I'm actually excited to go try this out
- 42:11and play around with it. I did find
- 42:13an article too on a Raspberry Pi cluster
- 42:15for AI. I don't know if I'm gonna
- 42:16try that or use an extra Mac mini
- 42:18I have sitting around here to start, but
- 42:20I, like you, I would love to hear
- 42:22what other people are doing. If you are
- 42:23running them locally, what are you using them
- 42:25for locally,
- 42:26different use cases,
- 42:28where have you found a good place to
- 42:29start, all the things.
- 42:31So if you do want to join us
- 42:33and discuss these things,
- 42:34we need to redo our outro, Scott, because
- 42:37I think that has changed. I think we
- 42:38actually still have Twitter in it. Let's not
- 42:40say Twitter. Let's say probably Blue Sky. Are
- 42:43you more active on Blue Sky right now
- 42:44than any other one? Pick one anyway. Anyone
- 42:46that's not Twitter, you can find Scott on,
- 42:48except that I can never find you on
- 42:49Blue Sky because you chose a weird handle
- 42:51that isn't the same as any of your
- 42:53other social media. You need to go grab
- 42:55a new handle on Blue Sky that matches
- 42:57everything else. I would say Blue Sky is
- 42:59probably where I'm the most active
- 43:02as of late and where I feel like
- 43:04the biggest
- 43:05tech community has
- 43:07moved to. So go chat with us on
- 43:09Blue Sky. LinkedIn is another good one. I'm
- 43:11always on LinkedIn.
- 43:13So if you wanna chat, give us feedback
- 43:15on LinkedIn,
- 43:16you can do that. If you wanna sign
- 43:17up for membership, we still have our membership
- 43:19at mscloud, I t pro Com / membership.
- 43:22Todd's in
- 43:23Discord today. He got a new laptop that
- 43:25he's gonna go try to run some LLMs
- 43:27on. So if you wanna join us, chat
- 43:29with us during the recording. You can go
- 43:31check out our membership
- 43:32options
- 43:33there as well and join us in Discord
- 43:36for these. So
- 43:37looking forward to hearing from people how you
- 43:40use LLMs, what you're gonna do with LLMs,
- 43:42and how they run locally. Who can bury
- 43:45their computer first and
- 43:48crash it? Super easy to do. Yeah.
- 43:51Anything else? I think that's it. As always,
- 43:53thanks, Ben. Alright. Thank you, Scott. We will
- 43:55talk to you later.
- 43:58If you enjoyed the podcast, go leave us
- 44:00a five star rating in iTunes. It helps
- 44:03to get the word out so more IT
- 44:04pros can learn about Office three sixty five
- 44:06and Azure.
- 44:08If you have any questions you want us
- 44:09to address on the show, or feedback about
- 44:12the show, feel free to reach out via
- 44:14our website, Twitter, or Facebook.
- 44:16Thanks again for listening, and have a great
- 44:18day.