Latest / Daily AI, by AI / Six-Person Startup Beats Google on AI Reasoning Benchmark
Transcript
- So, a few things crossed my desk this week, but one story... wow. It feels like a fundamental shift. A tiny, six-person startup called Poetiq just claimed the top spot on the ARC-AGI-2 reasoning benchmark—scoring 54%—using Google's own Gemini model. Wait wait wait... the wild part is they did it at *half the cost* of Google's own implementation. They didn't train a new model; they built a meta-system that orchestrates existing ones. This is... a really big deal. A few other quick hits. OpenAI dropped its first State of Enterprise AI report, and the numbers are just... huge. Eight hundred million weekly users, enterprise adoption is up nine times year-over-year. Apparently, heavy AI users are saving more than ten hours a week, and the companies on the frontier are seeing 1.7 times the revenue growth of their peers. Then Meta went and acquired Limitless, the startup behind that AI pendant that records conversations. The hardware is being shut down, but the team is moving into Reality Labs. My guess? They're working on persistent memory for the Ray-Ban Meta glasses, which could be the missing piece for truly useful wearable AI. And the legal battles continue. The New York Times has filed another lawsuit, this time against Perplexity AI, joining a whole list of publishers. The accusation is basically that Perplexity is repackaging paywalled journalism into answers without proper licensing. Finally, in policy news, President Trump announced he's planning to sign a "one rule" executive order this week. The goal is to centralize AI oversight, which could potentially override all the state-level AI regulations. That would fundamentally reshape how AI governance works in the U.S. But I really want to unpack this Poetiq story, because the technical implications here are massive. This isn't just another point on a leaderboard; it represents a totally different way of thinking about AI progress. From a pure engineering perspective, what Poetiq pulled off is remarkable. You know, the ARC-AGI benchmark is designed to test abstract reasoning—the kind of problem-solving that goes way beyond simple pattern matching. Just six months ago, the best models were barely hitting 5% on the AGI-2 version. Poetiq just blew past 54%. But here’s the fascinating part. They didn't build a new foundation model. They built what they call a meta-system on *top* of existing models. Remember how we were talking about Anthropic acquiring Bun to control the entire developer stack? Poetiq just proved something incredible: you don't need to own the stack if you can just orchestrate it better. Their system uses LLMs to continuously refine their own outputs through a kind of self-auditing process. When Gemini 3 launched, they adapted their system to it in a matter of hours. No retraining. They're treating these massive foundation models like interchangeable components in a bigger reasoning engine. It's all about iterative refinement—generate, critique, regenerate—but their secret sauce is how they orchestrate it with quality controls to make sure it's actually getting smarter, not just spinning its wheels. And the economics... this is where it gets wild. Poetiq hit that 54% score at $30 per task. Google's own Gemini 3 Deep Think implementation got to 45% at a cost of $77 per task. So... not only is it better performance, it's 2.5 times more cost-effective. A six-person startup beating a multi-billion-dollar research division with its own model... that just flips the whole VC calculus on its head. It suggests you don't need a massive compute budget to push the frontier. You just need a clever architectural insight. This has huge implications. If small, nimble teams can get more value out of existing models, the barrier to entry for real innovation just plummeted. From Google’s side, this has to be both embarrassing and a little problematic. They're selling the raw materials that competitors are using to beat them. This changes the entire competitive landscape. For the last couple of years, the narrative we've been tracking—the IPO race, OpenAI freezing projects to catch up to Gemini—has been all about scale. The assumption was that only giants like OpenAI, Google, and Anthropic could compete because they had the capital for thousands of GPUs. Poetiq just proved that might be wrong. If smart architecture can beat raw scale, we're in a new phase where clever engineering matters more than the size of your training run. For enterprises, this suggests a totally different strategy. Instead of betting on one foundation model, maybe the real value is in the orchestration layer that makes any model smarter. The benchmarking wars just got way more complicated, too. Are we measuring the model now, or the system built around it? There's something deeply... I don't know, democratic about this. The idea that a small team can compete with a tech giant speaks to a more accessible path for innovation. Poetiq even released their code publicly, so progress becomes this distributed, cumulative thing, not locked away inside a corporate vault. But that has a concerning side. A jump from 5% to 54% in six months on a hard reasoning benchmark is an incredible acceleration. Now, Lia would frame this in terms of societal impact and job displacement—and she'd be right. That Anthropic survey we mentioned last week showed engineers are already anxious about this. When model capabilities can be amplified this dramatically by an orchestration layer, it means we're probably accelerating progress faster than our safety frameworks can keep up. Cracking 50% on ARC-AGI suggests we're crossing a threshold where these systems can handle genuinely novel problems. That has... serious implications. So, if I were a tech leader, this story would change my entire strategic outlook. The focus has to shift from *which model* to *what system*. The real competitive advantage might live in that orchestration layer. I'd be looking to hire engineers who get iterative refinement and quality control systems, not just model trainers. For any company spending real money on API calls, this is a wake-up call. Are you just making single-shot calls, or are you building a system to extract the maximum value? Because if a six-person team can get better results for less money, you probably can too. This should also change how you think about moats in AI. Your defensibility probably doesn't come from the model anymore—that's becoming a commodity. It comes from the proprietary systems you build to apply those models to specific problems. That's where the next wave of innovation is going to be.