Latest / Ctrl AI Profit / Ep. 134 | Running Local AI Is Good Now — And Your Wallet Will Thank You
Transcript
- 0:20SPEAKER_00: There's a blog post blowing up right now called Running Local Models is Good Now.
- 0:24SPEAKER_00: And that headline alone should matter to every small business owner paying AI subscription fees.
- 0:31SPEAKER_01: The post is by Vicka Boykis, and it's been sitting at the top of Hacker News.
- 0:35SPEAKER_01: The thesis is simple.
- 0:37SPEAKER_01: Local AI models, the ones that run on your own hardware, not in the cloud, have crossed the usability threshold for real work.
- 0:46SPEAKER_00: Which means the you need the cloud narrative that every AI company has been selling might not be true anymore.
- 0:52SPEAKER_01: And if it's not true, that changes the economics of AI for small businesses dramatically.
- 0:58SPEAKER_01: Because right now, the cloud tax is real.
- 1:01SPEAKER_01: Every query costs something, every employee needs a license.
- 1:04SPEAKER_01: It adds up fast.
- 1:06SPEAKER_01: Let's be specific about what changed.
- 1:08SPEAKER_01: Six months ago, running a local model meant sacrificing quality.
- 1:12SPEAKER_01: The models were slow, dumb, and not worth the hassle.
- 1:16SPEAKER_01: That's no longer the case.
- 1:18SPEAKER_01: What changed?
- 1:20SPEAKER_01: Three things.
- 1:21SPEAKER_01: First, the models got way better at smaller sizes.
- 1:25SPEAKER_01: You can run a model that performs like last year's top-tier cloud model on a decent laptop.
- 1:30SPEAKER_01: Second, the tooling improved dramatically.
- 1:33SPEAKER_01: Apps like LM Studio and Alama made installation a one-click process.
- 1:39SPEAKER_01: And third, Apple Silicon and new Qualcomm chips gave consumer hardware enough muscle to run these efficiently.
- 1:46SPEAKER_00: So I can run AI on my laptop and it actually works.
- 1:51SPEAKER_01: For a lot of tasks, yes, you're not training a foundation model locally, but for writing, summarizing, data extraction, customer email drafts, basic code help, the stuff most small businesses actually need, local models handle it fine.
- 2:07SPEAKER_00: And here's the wallet impact.
- 2:09SPEAKER_00: Right now, small businesses pay per token or per month for Cloud AI.
- 2:13SPEAKER_00: ChatGPT Plus is 20 a month per user.
- 2:16SPEAKER_00: Enterprise plans run hundreds per seat.
- 2:19SPEAKER_00: A local model running on a machine you already own, zero additional cost per month.
- 2:26SPEAKER_01: Zero.
- 2:26SPEAKER_01: The hardware you have, the electricity to run it, and that's it.
- 2:31SPEAKER_01: No per token charges, no monthly subscriptions, no surprise bills when someone writes a long prompt.
- 2:38SPEAKER_01: Think about that.
- 2:39SPEAKER_00: Every prompt your team runs is free.
- 2:41SPEAKER_00: There's no meter running, no watching the token count climb.
- 2:45SPEAKER_00: You just use it as much as you want.
- 2:47SPEAKER_00: Real numbers.
- 2:49SPEAKER_00: Five people on ChatGPT Plus at 20 each.
- 2:52SPEAKER_00: That's$100 a month,$1,200 a year.
- 2:55SPEAKER_00: Local model, zero.
- 2:57SPEAKER_00: That's not a small difference.
- 2:58SPEAKER_00: That's a budget line item that disappears entirely.
- 3:02SPEAKER_01: Now, to be fair, there are trade-offs.
- 3:05SPEAKER_01: You need a decent machine.
- 3:07SPEAKER_01: The models aren't as capable as the biggest cloud models for complex reasoning.
- 3:11SPEAKER_01: And setup takes a couple hours if you're technical, maybe a day if you're not.
- 3:16SPEAKER_00: But here's what I find interesting.
- 3:17SPEAKER_00: Most small businesses aren't using AI for complex reasoning.
- 3:21SPEAKER_00: They're using it for the everyday stuff: drafts, summaries, data extraction, customer responses.
- 3:27SPEAKER_00: And for those tasks, a local model is plenty.
- 3:31SPEAKER_01: The performance gap is real but shrinking.
- 3:34SPEAKER_01: A local 7 billion parameter model today performs roughly like GPT 4 did two years ago.
- 3:40SPEAKER_01: And GPT 4 was considered cutting edge then.
- 3:44SPEAKER_01: For standard business tasks, that's more than enough.
- 3:47SPEAKER_00: And the privacy angle is enormous.
- 3:50SPEAKER_00: Local model means your data never leaves your machine, no API calls, no data sent to OpenAI's servers, no chance of your customer information ending up in a training data set.
- 4:02SPEAKER_01: For businesses handling sensitive data, medical records, financial information, legal documents, that's not a nice to have.
- 4:11SPEAKER_01: It's a requirement.
- 4:12SPEAKER_01: Cloud AI can't guarantee that level of privacy, period.
- 4:16SPEAKER_00: I know some businesses that won't touch cloud AI specifically because of data privacy concerns.
- 4:22SPEAKER_00: They've been sitting out the whole AI revolution.
- 4:25SPEAKER_00: Local models are there on ramp.
- 4:28SPEAKER_01: That's a really important point.
- 4:30SPEAKER_01: There are thousands of small businesses, especially in healthcare, finance, legal, that have been holding back from AI entirely because they can't risk putting client data through a cloud API.
- 4:41SPEAKER_01: Local models remove that barrier completely.
- 4:44SPEAKER_00: So how does a small business owner actually start?
- 4:47SPEAKER_00: Because install a llama sounds easy when a developer says it.
- 4:51SPEAKER_00: For most business owners, that's not a sentence they understand.
- 4:55SPEAKER_01: Here's the simplest path.
- 4:57SPEAKER_01: Download an app called LM Studio.
- 4:59SPEAKER_01: It's free, it has a graphical interface, no command line.
- 5:03SPEAKER_01: You pick a model from their library, click download, and start chatting with it.
- 5:08SPEAKER_00: As easy as installing any other app.
- 5:10SPEAKER_00: And once it's running, you type questions and get answers like ChatGPT.
- 5:16SPEAKER_01: Exactly.
- 5:16SPEAKER_01: Same chat interface, same back and forth.
- 5:20SPEAKER_01: The only difference is the model runs on your machine and your data stays on your machine.
- 5:24SPEAKER_00: No account to create, no credit card to enter, no company tracking your usage.
- 5:29SPEAKER_00: That's a different relationship with AI entirely.
- 5:33SPEAKER_00: What about speed?
- 5:35SPEAKER_00: People love that cloud AI is instant.
- 5:38SPEAKER_01: On a modern Mac with Apple Silicon, a 7 billion parameter model generates text nearly as fast as you can read it.
- 5:45SPEAKER_01: On a Windows machine with a recent GPU, same story.
- 5:49SPEAKER_01: Not instant, but fast enough that you won't be waiting.
- 5:52SPEAKER_00: And here's the strategic play.
- 5:54SPEAKER_00: Start with local models for the simple, repetitive tasks.
- 5:57SPEAKER_00: Keep cloud AI for the complex stuff.
- 6:00SPEAKER_00: You just cut your cloud bill by 50 or 60% without losing any capability.
- 6:05SPEAKER_01: That's the hybrid approach, and it's where smart businesses are heading.
- 6:09SPEAKER_01: Local for volume, cloud for complexity.
- 6:12SPEAKER_01: The cloud providers know this too.
- 6:14SPEAKER_01: It's why they're pushing enterprise features and agents.
- 6:17SPEAKER_01: They know the simple stuff is going local.
- 6:19SPEAKER_00: Which explains a lot.
- 6:21SPEAKER_00: OpenAI and anthropic both investing in consulting and ecosystems.
- 6:26SPEAKER_00: If models become commoditized and runnable locally, their value shifts to implementation, not the model itself.
- 6:34SPEAKER_01: Exactly, the model layer is being commoditized, the value is moving to how you use it, not which one.
- 6:41SPEAKER_01: Local models accelerate that shift because they break the paper token dependency.
- 6:47SPEAKER_00: Let me give a concrete example.
- 6:49SPEAKER_00: A local accounting firm, they handle sensitive client financial data.
- 6:53SPEAKER_00: They can't use cloud AI for any of it.
- 6:56SPEAKER_00: But with a local model, they can automate document summarization, draft client emails, extract data from tax forms, all without a single byte leaving their office.
- 7:06SPEAKER_01: That firm was sitting out AI entirely.
- 7:10SPEAKER_01: Now they're not.
- 7:11SPEAKER_01: That's the market that local models unlock.
- 7:13SPEAKER_01: Businesses that had legitimate reasons to avoid cloud AI now have a path in.
- 7:18SPEAKER_00: And that firm doesn't pay a dime in subscription fees.
- 7:21SPEAKER_00: They use hardware they already have, they get the same productivity gains everyone else is bragging about, without the privacy risk, without the monthly bill.
- 7:29SPEAKER_01: The bottom line: running local AI is no longer a hobby for tech enthusiasts.
- 7:34SPEAKER_01: It's a legitimate business strategy for cutting costs, protecting data, and reducing dependency on cloud vendors who are raising prices every quarter.
- 7:44SPEAKER_00: Start with LM Studio or Alama.
- 7:47SPEAKER_00: Download a model like Llama or Mistrol.
- 7:50SPEAKER_00: Try it on one task you're currently paying a cloud service for.
- 7:53SPEAKER_00: If it works, and for most standard tasks it will, you just found a permanent cost saving.
- 7:59SPEAKER_01: The cloud isn't going away, but its monopoly on AI access is.
- 8:04SPEAKER_01: And that's good news for anyone watching their AI bill climb every month.
- 8:08SPEAKER_00: And it's good news for businesses that have been sitting on the AI sidelines because of privacy.
- 8:14SPEAKER_00: Local models are their ticket in.
- 8:16SPEAKER_00: The cloud sold us on the idea that we need them.
- 8:19SPEAKER_00: For a lot of what we actually do, we don't.
- 8:22SPEAKER_00: That's the shift every small business owner should at least test.
- 8:26SPEAKER_01: Your laptop is more powerful than you think, and the models that run on it are better than you've been told.
- 8:32SPEAKER_01: Time to find out for yourself.