Latest / AI Fire Daily / #525 Neil: Stanford STORM Make Claude Research Like 5 Experts
Transcript
- 0:00You ask Claude for a complex research report.
- 0:02It gives you a remarkably clean, confident answer.
- 0:06You read it, and it sounds perfectly logical,
- 0:08but, you know, you walk away wondering if you
- 0:11can actually trust it. Because no one challenged
- 0:13the answer before handing it to you. Read. Yeah,
- 0:17the AI just patches together a very smooth narrative.
- 0:21It naturally avoids any friction or conflicting
- 0:23data points. Welcome to today's Deep Dive. We
- 0:26are exploring a totally different approach for
- 0:28your workflow today. We're breaking down a really
- 0:30powerful method to analyze sources. It's heavily
- 0:33inspired by Stanford's Storm project. We'll explore
- 0:36what Storm actually is. Then we'll show you how
- 0:38to split Claude into five conflicting expert
- 0:41lenses. We will map their contradictions carefully.
- 0:43And finally, we'll verify the synthesized output
- 0:46together. Beat. I have to admit something early
- 0:49on here. I still wrestle with trusting AI summaries
- 0:51myself. It's just too easy to accept a confident
- 0:54sounding paragraph. Oh, we all do it. Because
- 0:56it saves us so much time, right? But that convenience
- 0:59comes with a massive hidden cost. You end up
- 1:02accepting assumptions that were never properly
- 1:04tested. Our mission today is to fix that exact
- 1:07problem. This is how you stop trusting fake AI
- 1:11confidence. But before we fix Claude's blind
- 1:13spots, we need some context. We need to understand
- 1:16the fundamental flaw in standard AI research.
- 1:20Right. Standard prompts just ask for a final
- 1:22draft immediately. Exactly. Stanford recognized
- 1:25this flaw and built Storm to fix it. Storm is
- 1:28a very long, very academic acronym. It stands
- 1:32for Synthesis of Topic Outlines through Retrieval
- 1:35and Multi -Perspective Question Asking. I am
- 1:37very glad they shortened that to Storm. Yeah,
- 1:38me too. It's quite a mouthful. But the core philosophy
- 1:41is actually beautifully simple. Better research
- 1:44always starts with asking much better questions.
- 1:46It does not start with writing a better draft.
- 1:49Storm focuses entirely on the prewriting stage
- 1:51of your research. That's the exact opposite of
- 1:54how we normally work. We usually just open the
- 1:56chat and demand an outline. We rush straight
- 1:57to the final written product. We just skip the
- 2:00entire discovery phase. We do. And the Stanford
- 2:03data shows exactly why rushing fails. Using this
- 2:07multi -perspective method yields incredible measurable
- 2:10results. Articles become 25 % better organized
- 2:14overall. That's huge. It is. This is compared
- 2:17to standard AI outline generation. So the logical
- 2:20flow is significantly stronger for the reader.
- 2:22Right. And the actual topic coverage becomes
- 2:2410 % broader. It finds interesting angles you
- 2:27would have completely missed otherwise. The system
- 2:29collects sources, asks questions, and builds
- 2:32an outline first. It does all this before any
- 2:34actual writing begins. We should clarify a very
- 2:37important distinction for you right now. We are
- 2:40applying Storm's core principles directly into
- 2:42your cloud chat. We are not using the official
- 2:45Stanford software today. We're just adapting
- 2:47their brilliant logic for everyday use. It makes
- 2:50the methodology accessible to anyone with an
- 2:53AI chat. Think of standard prompting, like asking
- 2:55one person for directions. They might confidently
- 2:58send you straight into a muddy ditch. Storm is
- 3:00like putting five grumpy locals in a room. You
- 3:02make them argue and agree on the best route.
- 3:05I love that. Those grumpy locals know all the
- 3:07hidden roadblocks. They know the clever shortcuts.
- 3:10They pressure test the route before you even
- 3:12start driving. Two -sec silence. So the core
- 3:15shift is changing how we ask rather than focusing
- 3:18on the drafting. Exactly. Generating better questions
- 3:21before the actual drafting even begins. Since
- 3:24better questions are the key, we need those grumpy
- 3:26locals. We need to figure out who is asking these
- 3:29questions. We have to define these experts carefully.
- 3:32The biggest flaw of a single prompt is surprisingly
- 3:34simple. And a prompt is just these specific text
- 3:37instructions you type into the AI. A single prompt
- 3:40always have massive, unavoidable blind spots.
- 3:44It only looks at the problem from one specific
- 3:46angle. Yeah, exactly. So we need to introduce
- 3:48multiple experts to Claude. You tell Claude to
- 3:51adopt five specific expert lenses. You give it
- 3:54the topic and your specific role. You also state
- 3:57the specific decision you're trying to make.
- 3:58This context stops the AI from giving generic,
- 4:02useless answers. It gives each expert lens a
- 4:05very clear, distinct job. Let's deeply examine
- 4:08these five default lenses for you. First, we
- 4:11have the practitioner lens. What is their main
- 4:12focus? They focus entirely on what works in the
- 4:15real world. They want to know how this idea is
- 4:17actually implemented. So they care about execution,
- 4:20not just high -level academic theory. They want
- 4:22to know if it survives contact with reality.
- 4:25Second is the academic lens. They act like a
- 4:28strict university professor grading a paper.
- 4:30They care deeply about research quality and strict
- 4:33evidence standards. They demand to see the actual
- 4:36peer -reviewed proof behind claims. They refuse
- 4:40to accept anecdotal evidence or flashy marketing
- 4:43spin. Right. Third is the skeptic lens. They
- 4:46push back incredibly hard on assumptions and
- 4:48weak claims. They are the ultimate devil's advocate
- 4:51in the room. I imagine the skeptic is constantly
- 4:54looking for the hidden catch. They want to know
- 4:56why a seemingly great idea might fail. Fourth
- 4:59is the economist lens. They focus heavily on
- 5:02cost, return on investment, and market logic.
- 5:05They meticulously follow the money to see if
- 5:07it makes sense. The economist lens stops us from
- 5:10loving ideas that bankrupt us. They keep the
- 5:12entire project grounded in financial reality.
- 5:15And fifth is the historian lens. They look at
- 5:18past patterns that are repeating in the present.
- 5:20They give us the crucial context of time and
- 5:22precedent. The historian prevents us from repeating
- 5:25the exact same past mistakes. They remind us
- 5:28that very few problems are entirely new. When
- 5:31five different perspectives study the exact same
- 5:33topic, sparks fly. They aggressively push back
- 5:37on each other's core assumptions. That applied
- 5:40pressure. makes the final output much more reliable.
- 5:43There is a crucial instruction you must remember
- 5:44here. You must tell Claude to report each lens
- 5:47completely separately. You must do this before
- 5:50moving on to the next step. If Claude combines
- 5:52them too early, it ruins the entire process.
- 5:55It completely defeats the purpose of the pressure
- 5:57test. You just get a bland, averaged -out summary
- 6:00instead. It basically waters down the specific
- 6:02insights of each grumpy local. Check if it gathers
- 6:05the distinct sources first. If it gives one combined
- 6:08answer, stop the generation immediately. Ask
- 6:11it to separate the five distinct lenses first.
- 6:13Save the grand synthesis for later. Two -sex
- 6:16silence. Why must they report independently first?
- 6:20Why not just ask for a summary right away? To
- 6:22stop the AI from blending them into a bland compromise
- 6:26too early. So we have five highly opinionated
- 6:29expert reports. They are just sitting there in
- 6:31the chat, completely separated. What happens
- 6:34when they inevitably disagree with each other?
- 6:36Because if they are doing their jobs, they will
- 6:39definitely fight. Step two is where the real
- 6:41analytical magic happens. You push Claude to
- 6:44carefully map where the lenses disagree. You
- 6:47do this by sending a specific new prompt. We
- 6:50want to clearly see the friction between the
- 6:52experts. We are looking for the exact points
- 6:54of contention. Exactly. Instead of accepting
- 6:56each lens at face value, you probe them. Ask
- 7:00which underlying assumptions are actively contradicting
- 7:02each other right now. Ask which specific lens
- 7:04has the weakest supporting evidence overall.
- 7:07Wait, but if I am forcing Claude to map these
- 7:10disagreements... Doesn't that just confuse the
- 7:13AI completely during the process? Like, if the
- 7:15economist says a project is just too expensive,
- 7:17but the practitioner says we absolutely have
- 7:19to do it anyway, how does the AI not just crease
- 7:22up or water it down? That is the genius of separating
- 7:25the mapping from the synthesis. You are not asking
- 7:28for a final decision right now. You are just
- 7:31asking it to highlight the friction points clearly.
- 7:34Why do these specific disagreements matter so
- 7:36much to the listener? They signal weak assumptions
- 7:39that are cleverly hidden in the data. They highlight
- 7:42unverified claims that sound very good initially.
- 7:45They expose fragile conclusions that fall apart
- 7:47under new evidence. Do not let Claude casually
- 7:50skip over the small conflicts. Make it flag every
- 7:53single point of conflict clearly. In practice,
- 7:57that is exactly where the biggest risks hide.
- 8:00This step makes the research infinitely more
- 8:03useful for you. You can clearly see which ideas
- 8:05are actually structurally sound. You see which
- 8:08ones need much more evidence before trusting
- 8:10them. So we've got these five experts arguing
- 8:12in a digital room. They're pointing out each
- 8:14other's hidden blind spots. But at the end of
- 8:16the day, I still need one cohesive report. How
- 8:20do we take that messy debate and synthesize it
- 8:22cleanly? Now we finally move to step three, synthesizing
- 8:26everything into one unified structured report.
- 8:29Now, Claude stops acting like five separate argumentative
- 8:33experts. It becomes one unified, highly rational
- 8:36voice for you. But there is a crucial weighting
- 8:38rule here for the synthesis. Conclusions supported
- 8:41by multiple lenses carry much more weight. That
- 8:44keeps the final report properly focused and grounded.
- 8:47It stops it from treating every single finding
- 8:49as equally important. If the academic and the
- 8:51practitioner agree, That is gold. Where lenses
- 8:54conflict, it must present both sides objectively,
- 8:57then have Claude explain which side actually
- 8:59has stronger evidence. This makes the final answer
- 9:02much easier to trust. Stronger ideas are cleanly
- 9:06separated from weaker, unsupported opinions.
- 9:08To sex silence. Do we completely discard the
- 9:11ideas that only one lens supported during synthesis?
- 9:14No, just set them aside for deeper review before
- 9:16trusting them. We will be right back after a
- 9:18quick word. Mid -roll sponsor break. And we are
- 9:21back. Even with a carefully weighted synthesis,
- 9:24things can get tricky. AI can still hallucinate
- 9:27confidence very easily on complex topics. And
- 9:30hallucination is just when AI confidently presents
- 9:33false information as fact. How do we ensure the
- 9:36draft is actually grounded in reality? Step four
- 9:38is the crucial verify step. This is where we
- 9:41double check the maps of our grumpy locals. You
- 9:44force Claude to fact check the entire synthesized
- 9:46draft. You make it meticulously score all the
- 9:49original sources. It uses three specific labels
- 9:52for this rigorous process. First is the confirmed
- 9:54label. This means the source fully and demonstrably
- 9:57supports the claim. Second is the corrected label.
- 10:01This means the original claim needs adjustment
- 10:03based on actual facts. The AI essentially caught
- 10:06itself stretching the truth. Third is the demoted
- 10:08label. The source just isn't strong enough to
- 10:10back the claim up. Maybe it was just an opinion
- 10:11piece, not actual research. This verification
- 10:14step is absolutely vital for your ultimate trust.
- 10:17It stops the report from sounding overly confident,
- 10:19especially when it is actually resting on very
- 10:22weak evidence. The output structure makes this
- 10:24incredibly easy to read. It starts with a clean,
- 10:2760 -second executive summary. This is perfect
- 10:30for anyone who is desperately short on time.
- 10:32Yeah, it's super helpful. Then it logically lists
- 10:34the key findings ranked by reliability. Next,
- 10:38it clearly shows the reliability scores for each
- 10:40distinct point. Finally, it gives you a deeply
- 10:43detailed source review. Here is a pro tip for
- 10:46your workflow. Always read those reliability
- 10:48scores before you read anything else. Conclusions
- 10:51heavily backed by multiple lenses are generally
- 10:53safe to use. Let's bring in some real -world
- 10:56examples for this process. We need to prove exactly
- 10:59why this rigorous method matters. Let's deeply
- 11:01examine the AI wage premium claim. A standard
- 11:04prompt might boldly say there's a 43 % premium.
- 11:07It sounds very specific and highly believable
- 11:10at first glance. But the verify step actually
- 11:12corrects this specific claim. It forces the AI
- 11:15to look closer at the underlying methodology.
- 11:18It reveals PwC's actual calculated figure is
- 11:2156 % and they base that on nearly 1 billion job
- 11:25postings. The 43 % number actually comes from
- 11:28Lightcast instead. That specific number measures
- 11:31a much narrower set of criteria. But the two
- 11:34completely different figures get mixed up constantly.
- 11:37Beat. Whoa. Imagine scaling to analyze nearly
- 11:41a billion job postings just to find that exact
- 11:43premium. It is a truly staggering amount of data
- 11:46to process. But it shows why finding the exact
- 11:50source matters deeply. Another excellent example
- 11:53is the AI agent market size. A standard AI might
- 11:56claim it's exactly $103 .6 billion. it presents
- 12:01that massive number as an absolute undeniable
- 12:04fact. But the verify step corrects this to a
- 12:07much broader range. Depending on the methodology,
- 12:09the range is actually massive. Different credible
- 12:12sources put it between 52 billion and 183 billion.
- 12:15This is a massive flashing warning sign for you.
- 12:18Using a single definitive figure without a range
- 12:21is deeply dangerous. That is exactly how rampant
- 12:24misinformation spreads in AI content. You can
- 12:26also selectively swap out these expert lenses.
- 12:28After reading the draft, ask yourself which perspective
- 12:30is missing. Maybe the default five skipped the
- 12:33actual reader side You can easily swap a default
- 12:36lens for something else. Try using beginner,
- 12:39content creator, or even end user. Just update
- 12:41the step one prompt and run it all again. Two
- 12:44secs silence. So a corrected label doesn't mean
- 12:47the AI failed. It just means the data is messy.
- 12:50Right. It forces you to see the nuance instead
- 12:52of a fake absolute. Let's summarize the core
- 12:55philosophy for you today. One standard prompt
- 12:57gives you one single narrow angle. That just
- 13:01isn't enough to hold up under serious pressure.
- 13:04It is like asking one person for critical directions.
- 13:08By using Storm's multi -perspective approach,
- 13:10everything fundamentally changes. You research,
- 13:13you map contradictions, you synthesize, and crucially,
- 13:16you verify. You actively stop weak evidence from
- 13:19surviving the final cut. It stops weak evidence
- 13:22from being dressed up in confident language.
- 13:23You essentially force the AI to do the incredibly
- 13:26hard work. You make it argue with itself before
- 13:28it answers you. Yeah, you build a resilient answer.
- 13:30If we can instantly spawn a panel of five virtual
- 13:33experts to debate any topic, it changes how we
- 13:36learn. But here is something to chew on. If we
- 13:38rely entirely on AI to map the contradictions
- 13:41of human knowledge, do we risk losing our own
- 13:43ability to spot the blind spots? Try running
- 13:45a topic you know deeply through the Storm workflow
- 13:47this week. See if the AI's experts catch the
- 13:50nuances you already know are there. OTRO music.