Latest / AI Fire Daily / 🎙️ EP 313: Thinking Machines Drops "Inkling" & PrismML Fits a 27B Model on Your Phone
Transcript
- 0:00For the last two years, I feel like we've been
- 0:01chasing the biggest, most godlike AI models in
- 0:06the cloud. Yeah, totally. But I was thinking,
- 0:08what if the future of AI isn't in a massive data
- 0:12center, but sitting right in your pocket? Right,
- 0:14because today, Miramarati just dropped a 975...
- 0:18billion parameter model. And somehow the industry
- 0:22is simultaneously figuring out how to cram that
- 0:25exact level of intelligence directly onto your
- 0:27smartphone. It was kind of wild. Plus, we are
- 0:29tracking some really intense ecosystem drama
- 0:32right now between Microsoft, Apple and OpenAI.
- 0:35We really are. Welcome to the Deep Dive, everyone.
- 0:39We are seeing a massive pivot in the entire AI
- 0:41landscape today. We really are. We're basically
- 0:44moving from a single generic intelligence to
- 0:46highly customized, hyper -efficient models. So
- 0:49let's just dive straight into the biggest news
- 0:51for you. Yeah, let's do it. Miramarati's new
- 0:53venture, Thinking Machines Lab, just released
- 0:55their first open -weight model. It's called Inkling.
- 0:59And it represents a fundamental departure from
- 1:01the strategy of, you know, her former employer,
- 1:04OpenAI, because they are not trying to build
- 1:07one godlike model to rule them all. They are
- 1:10building a foundational base for extreme customization.
- 1:14Yeah. And on paper, Inkling sounds like a traditional
- 1:17behemoth. Right. I mean, it's a 975 billion parameter
- 1:21Moe system. Exactly. And it was trained on, what,
- 1:2545 trillion multimodal tokens? Yeah. $45 trillion.
- 1:29Which covers text, images, audio, and video.
- 1:32But that massive parameter count is... Kind of
- 1:35deceptive. How so? Because of how that MOE architecture
- 1:37actually functions in practice. Right. Let's
- 1:40clarify that term for a second. MOE stands for
- 1:43a mixture of experts. It's basically a model
- 1:46that only uses parts of its brain when needed.
- 1:48Think of it like a massive hospital. You don't
- 1:50need the heart surgeon, the neurologist, and
- 1:52the physical therapist all in the room just to
- 1:54look at a sprained ankle. No, that would be overkill.
- 1:56Exactly. The model's routing algorithm looks
- 1:58at your prompt and sends it only to the specific
- 2:01neural networks, the experts that actually know
- 2:03how to handle it. Which means out of 975 billion
- 2:06parameters, Inkling only wakes up about 41 billion
- 2:09for any specific task. Yeah. It keeps the compute
- 2:12costs incredibly low. But what really caught
- 2:15my eye is how Inkling fundamentally alters the
- 2:18problem -solving process itself. Oh, you're talking
- 2:21about the uncertainty flags. I am, yeah. Because
- 2:23we usually expect models to just confidently
- 2:25output text, even if they are completely hallucinating.
- 2:29Right. They just guess. Exactly. Yeah. But Inkling
- 2:32calculates a mathematical confidence threshold
- 2:34before it speaks. Yes. So if the statistical
- 2:37probability of the next token being correct falls
- 2:39below a certain internal benchmark, the model
- 2:42simply stops. It just pauses. Right. It flags
- 2:45its own uncertainty. It will actually prompt
- 2:47the user for clarifying context rather than just,
- 2:50you know, blindly guessing and outputting. confident
- 2:52hallucination. That is a massive architectural
- 2:55shift for Enterprise Trust. And it also features
- 2:58a physical dial for what they call thinking effort.
- 3:01Oh, the dial is so cool. You can actively trade
- 3:03speed for deep computation. So how does that
- 3:07work in practice? Well, if you ask it a simple
- 3:09factual question, you dial the effort down and
- 3:12it fires back instantly. But if you ask it to
- 3:15architect a complex database, you turn the dial
- 3:17up. The routing algorithm allocates more active
- 3:21parameters and, you know, really takes its time
- 3:24to reason. And the efficiency metrics are staggering
- 3:27here. On standardized coding benchmarks, Inkling
- 3:30matched NVIDIA's top open weight model. Yeah.
- 3:34But it achieved that exact same performance using
- 3:36just a third of the active tokens. Because the
- 3:39expert routing is just that precise. But, you
- 3:42know, thinking machines isn't just giving this
- 3:44away for free. Right. They have a business model.
- 3:46Yeah. Their entire business model hinges on a
- 3:48secondary platform they launched alongside Inkling,
- 3:51which is called Tinker. Right, and Tinker is
- 3:53where the customization happens. It's like stacking
- 3:56Lego blocks of data. Exactly. You take the highly
- 3:58efficient inkling -based model, and you use Tinker
- 4:01to fine -tune it exclusively on your company's
- 4:04private data. You basically freeze the foundational
- 4:06weights of the model so you don't break its general
- 4:09understanding of the world. Right. Then you train
- 4:11a very small, highly specialized adapter layer
- 4:14over the top using your proprietary data. It
- 4:17is fast, and it is incredibly cheap. Okay, but
- 4:20let me ask you this. If custom is cheaper and
- 4:23better, why is anyone still buying generic models?
- 4:27Because generic models are a fantastic starting
- 4:30line. I mean, if you're a startup, they are great.
- 4:32Sure. But specialized knowledge mathematically
- 4:35beats general knowledge in an enterprise setting.
- 4:38Like, look at the recent benchmark from that
- 4:40major hedge fund. Oh, right. They tested a customized,
- 4:43fine -tuned model against a top -tier, closed
- 4:47API model on complex financial logic. And the
- 4:51custom model didn't just win, it won while costing
- 4:53fractions of a cent per query compared to dollars
- 4:57per API call for the generic model. Exactly.
- 5:00The generic model got lost in basic Python boilerplate,
- 5:03while the custom model leveraged 10 years of
- 5:05proprietary quantitative data. So for tight security
- 5:08and hyper -specific logic, custom AI simply wins
- 5:12the marathon every time. So generic is the appetizer,
- 5:15but custom AI is the actual main course. That
- 5:17is exactly how the enterprise market is treating
- 5:19it right now. Well, customizing a model in the
- 5:21cloud is great for a hedge fund. But if the goal
- 5:24is absolute data privacy... You can't be sending
- 5:27sensitive data to external servers at all. No,
- 5:29definitely not. You need the model in the room
- 5:31with you. Yeah. Which brings us to a, frankly,
- 5:34disruptive release from Prism ML. Yes. They just
- 5:37dropped Bonsai 27B, and it fundamentally changes
- 5:41where high -level artificial intelligence can
- 5:43physically live. Right, because usually... A
- 5:4627 billion parameter model is enormous. I mean,
- 5:49even compressed, it sits around 18 gigabytes.
- 5:52And smartphones physically cannot handle that.
- 5:55They really can't. An average smartphone might
- 5:57have... 8 gigabytes of total unified memory.
- 6:00The operating system needs 3 gigabytes. Your
- 6:03background apps take another 2 gigabytes. So
- 6:06you're already almost maxed out. Exactly. If
- 6:08an AI app suddenly asks for 18 gigabytes of RAM,
- 6:11the phone's architecture just brutally shuts
- 6:13the process down to prevent a crash. But Bonsai
- 6:1627B is somehow only 3 .9 gigabytes. Yeah. They
- 6:19solved this physical physics problem through
- 6:21an extreme version of a process called quantization.
- 6:24Let's define that real quick. Quantization is
- 6:27shrinking file size by making the math slightly
- 6:29less exact. Perfect way to put it. But going
- 6:32down to 3 .9 gigabytes means they are using one
- 6:35-bit quantization. How does a model lose that
- 6:38much mathematical precision and still function?
- 6:41It is just brilliant engineering. Normal AI models
- 6:44use highly precise 16 -bit decimal numbers for
- 6:48their weights. Right. One -bit quantization compresses
- 6:51those complex decimals into simple binary choices,
- 6:54a 1 or a negative 1. But doesn't that destroy
- 6:57the model's ability to reason? I mean, you're
- 6:59losing so much data. You would totally think
- 7:01so. But neural networks are massively redundant.
- 7:04Oh, interesting. Yeah. The architecture actually
- 7:06compensates for the lost precision in individual
- 7:08parameters by relying on the aggregate strength
- 7:11of the network. It loses exact mathematical precision,
- 7:14but it keeps the overall logical structure entirely
- 7:17intact. And they released a few variants of this,
- 7:19right? Yeah. There is a ternary variant designed
- 7:22for laptops. That one sits at 5 .9 gigabytes
- 7:24and retains about 95 % of the original model's
- 7:27performance. And then there is the phone -ready
- 7:30version, the one -bit variant we just discussed.
- 7:323 .9 gigabytes, retaining 90 % of its performance.
- 7:36It handles multimodal inputs, complex coding,
- 7:40and even multi -step agent workflows. Wait, normally
- 7:43models this size need massive, power -hungry
- 7:45GPUs just to output a single word without lagging?
- 7:49True. Does this actually run fast enough on a
- 7:51standard smartphone to be usable? Or is it going
- 7:54to just melt my battery in 10 minutes? That is
- 7:56the beauty of modern mobile hardware. Smartphones
- 7:59now have dedicated NPUs, neural processing units.
- 8:02OK. Bonsai is specifically optimized to run entirely
- 8:05on that low power NPU, completely bypassing the
- 8:09main processor. So it sips battery instead of
- 8:11chugging it. Let's make this real for a second.
- 8:14Imagine you're on a flight. You're in airplane
- 8:16mode over the Atlantic. Zero Wi -Fi. Yeah. With
- 8:19BOSE 27B on your phone, you could snap a photo
- 8:22of a messy printed spreadsheet. And completely
- 8:25offline, your phone reads the image, writes a
- 8:27Python script to clean the data, executes the
- 8:29code locally, and hands you a perfectly formatted
- 8:31file. Whoa. Imagine running a 27 billion parameter
- 8:35brain completely in airplane mode. It's crazy.
- 8:39That completely shifts the power dynamic for
- 8:41the user. Total privacy. No API cost per query.
- 8:45Your data never leaves your hands. It bypasses
- 8:48the entire cloud subscription model. The model
- 8:51is fully open source and sitting on Hugging Face
- 8:53right now. Which raises an existential question
- 8:55for the big labs. Will this localized approach
- 8:58eventually kill off the cloud AI APIs entirely?
- 9:02Not entirely. I mean, the cloud will always be
- 9:04necessary for massive god -tier reasoning. Right.
- 9:07Like, if you are simulating a new pharmaceutical
- 9:09drug, you need a data center. But for writing
- 9:11emails, analyzing spreadsheets, and basic coding,
- 9:14Edge AI handles that beautifully. Cloud stays
- 9:17for heavy lifting, but our phones handle the
- 9:19daily grind locally. Precisely. And that push
- 9:22toward local control is being driven by a massive
- 9:24breakdown in enterprise trust right now. Which
- 9:26is the perfect lens to view the current drama
- 9:28unfolding across the tech ecosystem this week.
- 9:31We have to talk about the GrokBuild incident.
- 9:33Oh, man. Yeah. It is a stark reminder of why
- 9:35CIOs are terrified of cloud -based coding agents
- 9:38right now. GrokBuild was caught quietly uploading
- 9:42entire customer code repositories to company
- 9:45-controlled cloud storage. Proprietary enterprise
- 9:47code is the crown jewel of any tech company.
- 9:50Absolutely. Having an AI agent scrape that and
- 9:52pull it into a centralized cloud is a catastrophic
- 9:55security failure. SpaceX AI stepped in and claimed
- 9:59they're deleting the data, but the trust is already
- 10:02shattered. And competitors are ruthlessly leveraging
- 10:05that exact mistake. Look at Microsoft's leaked
- 10:08fiscal year 27 sales strategy. Yeah, that leak
- 10:12was fascinating. They are actively coaching their
- 10:14sales teams to knock down their own partners.
- 10:16Right. They're actively targeting open AI and
- 10:19anthropic in their enterprise pitches. They aren't
- 10:22selling raw intelligence anymore. They're pitching
- 10:24Microsoft Copilot is a heavily fortified enterprise
- 10:27bunker. They're telling CIOs that those standalone
- 10:30models are basically the Wild West and Copilot
- 10:32is the only compliant SOC2 certified environment
- 10:36safe enough for their data. So is Microsoft actively
- 10:39sabotaging OpenAI by calling them less safe?
- 10:42Well, it's classic enterprise strategy. CIOs
- 10:45buy data security, not just raw intelligence.
- 10:47They're selling a digital fortress, not just
- 10:49a faster horse. Exactly. It is a brilliant, if
- 10:54aggressive, enterprise strategy. Sponsor. And
- 10:57we are seeing this infrastructure fragmentation
- 10:59everywhere, heavily influenced by geopolitics,
- 11:02too. Like Apple intelligence is finally rolling
- 11:05out in China. But they can't use their Western
- 11:07partners there due to strict data regulations.
- 11:10Right. So Apple had to partner with. Alibaba
- 11:12using their Quen model to power the iPhone experience
- 11:15in Beijing. Which proves that local laws are
- 11:18now dictating global AI architecture. It really
- 11:21does. Meanwhile, the capital is flowing into
- 11:23hyper -specialized cloud applications that solve
- 11:26single, massive problems. Oh, like Chai Discovery.
- 11:29Yes, Chai Discovery just raised $400 million,
- 11:31hitting a $3 .8 billion valuation. And they aren't
- 11:35generating text or images. They built a specialized
- 11:38AI platform that strictly predicts how biological
- 11:41logical molecules interact with each other. Right.
- 11:43It accelerates antibody discovery by years. And
- 11:45while Tri is specializing in biology, OpenAI
- 11:48is pushing boundaries in physical hardware and
- 11:51internal security. They just released Codex Micro.
- 11:54The keyboard, right? Yeah, it's a $230 physical
- 11:56keyboard. It features dedicated agent keys and
- 11:59a physical reasoning dial. It basically takes
- 12:02this abstract cloud intelligence and anchors
- 12:04it into a highly tactile physical workflow. But
- 12:07you know, their most interesting release is actually
- 12:09invisible to users. You mean GPT -RED? Yeah,
- 12:12GPT -RED. It is an internal red teaming model.
- 12:15Its entire existence is dedicated to relentlessly
- 12:18attacking their own production models. AI attacking
- 12:21AI at scale. Exactly. It automates prompt. injections,
- 12:24firing millions of variations of ignore all previous
- 12:27instructions at the main system to find vulnerabilities.
- 12:30And it is working. The automated attacks help
- 12:33train their new GPT 5 .6 SAW model to achieve
- 12:37six times fewer security failures. That's huge.
- 12:40And speaking of new tools, the sheer volume of
- 12:43things dropping is just relentless right now.
- 12:45Seriously. We have Vela 3 .0 generating polished
- 12:48videos with localized translations. And V2Fun
- 12:51generating 8K 3D. characters from 2D images.
- 12:55Right, plus tools like Campus. Yeah, bringing
- 12:57your team into shared virtual workspaces. And
- 13:00agently doing workflow automation. And crest
- 13:02data, turning Claude into an automated recruiter
- 13:05scanning a billion profiles. It's crazy. But
- 13:08looking at this specific list of tools, I'm noticing
- 13:12a distinct narrative shift. We are moving away
- 13:15from AI as a simple chat interface. It seems
- 13:19like we're moving from AI that just chats to
- 13:22AI that acts as a localized coworker. That is
- 13:25the exact trend. If you look at a tool like Agently,
- 13:28it doesn't wait for you to prompt it. It acts
- 13:30as a central hub, connects to your apps, and
- 13:32automates complex multi -step workflows entirely
- 13:34on its own. And Campus takes that a step further.
- 13:37It brings your human team and your AI agents
- 13:40into one shared virtual office workspace. You
- 13:43aren't opening a siloed chat window anymore.
- 13:45The AI is sitting in the project files with you.
- 13:48But, you know, all of these autonomous co -worker
- 13:50style tools require a fundamental shift in how
- 13:52humans communicate with each other. That is very
- 13:54true. Which brings us to the new research paper
- 13:56from Sully AI. Right. A researcher there proved
- 13:59that the performance of an AI on complex tasks
- 14:01improves exponentially based purely on the style
- 14:04of your input. If you give an AI a casual prompt,
- 14:07like write me a marketing email, it defaults
- 14:10to average generic output. But if you provide
- 14:13a serious long horizon task brief, giving it
- 14:16a specific persona, strict constraints, and required
- 14:19formatting, the reasoning capabilities spike
- 14:22dramatically. I'll be honest, I still wrestle
- 14:25with prompt drift myself whenever I'm too casual
- 14:28with my AI. Oh, same here. I get lazy, I type
- 14:31a quick half thought, and then I find myself
- 14:33getting deeply frustrated. by the generic boilerplate
- 14:36answer it spits back. We all do it. We subliminally
- 14:39expect the machine to read our minds and infer
- 14:41our context. Right. But the data unequivocally
- 14:44shows that treating the model like a serious
- 14:46employee with a structured brief yields vastly
- 14:49superior results. Let's take a step back and
- 14:51connect the dots of this entire deep dive. Yeah,
- 14:53let's do it. We are clearly closing the chapter
- 14:55on the era of one massive generic oracle in the
- 14:58cloud answering everyone's questions. The new
- 15:01era we are entering is fiercely fragmented. It
- 15:03is highly specialized. And it is deeply personal.
- 15:06We see Inkling being fine -tuned exclusively
- 15:09for a single business's private data. We see
- 15:13Microsoft building walled enterprise gardens.
- 15:15And we see Bonsai 27B living right on your phone,
- 15:18completely severed from the cloud. Which raises
- 15:21an incredibly provocative question for you to
- 15:24consider this week. What's that? If our personal
- 15:26phones are now physically capable of running
- 15:2827 billion parameter models completely offline
- 15:32in airplane mode, what happens when those localized
- 15:36device -level AIs start communicating directly
- 15:39with each other? Wow. We might be standing on
- 15:42the very edge of a localized mesh network of
- 15:44intelligence, a peer -to -peer web of AI that
- 15:47bypasses the massive tech giants entirely. A
- 15:50decentralized swarm of highly capable models
- 15:53sharing context without ever touching a server
- 15:56in California. That is a staggering thought.
- 15:58It really is. As we wrap up today, I encourage
- 16:00you to rethink your own daily AI interactions.
- 16:02Try treating your AI to a serious, structured
- 16:05task brief instead of a casual prompt. See how
- 16:07much the output improves. And if your phone has
- 16:09the space, Maybe try downloading a local open
- 16:11source model like Bonsai to your hardware. Yeah,
- 16:14try it out. Experience exactly what true disconnected
- 16:17edge AI actually feels like. Because the future
- 16:20of AI isn't just humming in a massive data center
- 16:23anymore. It is sitting right in your pocket.
- 16:25Thank you for joining us for this deep dive.
- 16:27Ochi Aro Music.