Latest / AI Fire Daily / #4 Eric: Master Research Across the Google AI Ecosystem

Transcript

#4 Eric: Master Research Across the Google AI Ecosystem · AI Fire Daily

0:00
you're sitting in your kitchen right and you're
0:01
talking about buying a new espresso machine you
0:03
haven't searched for it you haven't typed it
0:05
anywhere beat and then 10 minutes later there's
0:08
an ad for it on your phone it's um It's the ultimate
0:12
modern paranoia. Are our phones actually listening
0:15
to us, or is something else happening beneath
0:18
the surface? It really is, yeah. It's the defining
0:20
mystery of our digital age. I mean, everybody
0:23
has a story exactly like that kitchen scenario.
0:26
So we actually used Google's most powerful AI
0:29
tools to investigate that specific paranoia.
0:32
And the reality of how those ads target you is,
0:35
well... It's so much stranger than your phone
0:38
simply recording your voice. But uncovering that
0:41
truth, it required a totally different approach
0:43
to research. Welcome to today's Deep Dive. We're
0:46
really glad you're here. Our mission today is
0:48
straightforward but vital. We want to dismantle
0:50
a very specific illusion. We're breaking the
0:52
illusion that all AI answers are created equal.
0:55
Because we tend to think AI just gives us, well,
0:57
the truth. Right. But that simply isn't how it
0:59
works. Not at all. So we're mapping out a completely
1:02
new journey today. We're going to break down
1:04
Google's five distinct research tools. We'll
1:07
reveal a bulletproof five -step workflow for
1:09
high -stakes research. And finally, we'll crack
1:12
open that listening phone experiment you mentioned.
1:15
It perfectly demonstrates how humans and AI absolutely
1:18
must collaborate. So let's start by looking at
1:20
the information landscape. Choosing the right
1:22
route is essential here. beat. It seems like
1:25
knowing which tool to pull from the toolbox is
1:27
basically half the battle, right? Like if I grab
1:30
the wrong tool, I'm gonna get the wrong kind
1:31
of answer. That is entirely true, yeah. Right
1:35
now, Google gives you five distinct options for
1:38
research. First, you have standard Google search.
1:41
You use this when you need to see the raw, unfiltered
1:43
landscape. You want to look around before any
1:46
AI summarizes things for you. Right, because
1:48
sometimes that raw data is the point. You want
1:50
to find original pages. You need to see recent
1:53
breaking news. Or maybe you just want to see
1:55
exactly who is talking about a specific topic
1:58
so you can evaluate their credibility yourself.
2:01
Mmm, exactly. Standard search lets you see the
2:04
original context. Second, you have Gemini. And
2:07
this is for when you just want the explanation
2:08
first. It's perfect for understanding a complex
2:10
topic quickly. You want a digestible summary
2:13
or maybe a fast conversation to just get your
2:16
bearings. But with Gemini, the AI has already
2:19
chosen what to surface. It curates the information
2:22
for you, so you see significantly less of the
2:24
surrounding evidence. Right. And that is the
2:26
fundamental trade -off, speed versus transparency.
2:29
So third is AI mode. This sits nicely in the
2:32
middle. It lets you explore broad questions,
2:34
but crucially, the open web is still visible.
2:37
It provides a web -grounded synthesis. Web -grounded
2:40
synthesis. Yeah, it gives you clickable links
2:42
right there in the text, so you can follow up
2:44
on claims easily. OK, so it's like selecting
2:46
a vehicle for a road trip. Standard Search is
2:49
a dirt bike for navigating raw terrain. Gemini
2:52
is a guided tour bus for quick sightseeing. And
2:55
AI Mode is a comfortable hybrid car. I love that
2:58
analogy. It's perfect. If you take the tour bus
2:59
off -roading, you're going to get stuck. Then
3:01
we get to the heavy hitters. Number four is Gemini
3:04
Deep Research. You use this when you need to
3:06
hand off a massive investigation. It just goes
3:08
out. digs for hours, and gives you structured
3:11
reports with citations. Let's define deep research
3:14
clearly for a second. It's essentially AI autonomously
3:17
finding, reading, and summarizing multiple web
3:19
pages for you. That hits the nail on the head.
3:22
Yep. And finally, number five is Notebook LM
3:25
or Gemini Notebook. This is entirely different.
3:28
It's for keeping your evidence in a closed, reusable
3:31
workspace. You bring in your own files or specific
3:34
websites. It relies entirely on a concept called
3:37
grounding. And grounding, just to clarify, that
3:39
means restricting the AI to only use the exact
3:42
documents you upload. Yes. Notebook LM keeps
3:45
you intimately close to the specific sources
3:47
you hand -selected. It won't pull random facts
3:50
from the broader internet. So extending the vehicle
3:52
metaphor. Deep Research is an autonomous, all
3:55
-terrain rover doing the exploring for you. And
3:58
Notebook LM is a high -tech laboratory where
4:00
you control all the variables. Right, a closed
4:02
system. But that raises a big question. If deep
4:05
research is so incredibly powerful, why not just
4:08
use it for absolutely everything? Well, because
4:10
handing off the entire investigation means you
4:12
lose intimate contact with the evidence. You
4:15
get a beautifully finished comprehensive report.
4:17
Sure. But you completely miss the messy nuance
4:21
of those raw sources. You don't see the contradicting
4:24
studies that might have just skimmed past. So
4:26
speed trades off with deep hands on control of
4:29
the facts. Precisely. If you're deciding where
4:32
to eat, use the fast tool. If you're building
4:35
a legal case, you need the control. And if you
4:37
want to maintain that deep control, you need
4:39
a system. Which brings us to the golden rule
4:42
of modern research. The golden rule. That sets
4:45
a high bar. What is it? It's a simple, reliable
4:48
shortcut, really. The more important the final
4:51
output is, the stronger your evidence standard
4:53
absolutely must become. That rule matters significantly
4:56
more than which specific software tool happens
4:58
to be open on your screen. Two -sex silence?
5:01
I have to admit, I still wrestle with treating
5:05
every AI citation as equally reliable. Just because
5:09
a paragraph has a little blue link next to it
5:11
doesn't mean it's gospel. Oh, it's a remarkably
5:13
common trap. AI systems are incredibly fast,
5:17
and they sound incredibly confident. But a 40
5:19
-page report does not guarantee the underlying
5:22
sources are actually strong. An AI will confidently
5:25
cite a random blog post from 10 years ago just
5:29
to fulfill its programming. That's exactly why
5:32
we need a rigorous five -step workflow for the
5:35
bigger jobs. So let's unpack that workflow. I'm
5:37
assuming we can't just type, tell me everything
5:39
about X, and hit enter. So the first step has
5:41
to be framing the problem correctly. You're spot
5:44
on. Step one is ask. You must explicitly define
5:47
the research job. You state exactly what you
5:49
want to understand, you define what the final
5:51
output should look like, and you clarify who
5:54
the intended audience is. And I imagine you must
5:56
keep the question strictly neutral. You don't
5:58
want to assume the answer in advance, or the
5:59
AI will just find evidence to confirm your bias.
6:02
Right. If you ask, why is coffee bad for you?
6:06
Hey, the AI will find you 100 reasons why it's
6:09
toxic. If you ask, what are the health impacts
6:11
of coffee? You get a balanced view. So step two
6:15
is research. This is where you actually build
6:17
the evidence pool. You might use deep research
6:20
here to cast a massive net and gather 50 potential
6:23
sources. But gathering sources isn't the same
6:26
as trusting them. We just established that a
6:28
lot of those sources might be garbage. So the
6:30
next logical hurdle is figuring out how to stop
6:32
the AI from using the bad ones. Which leads us
6:35
perfectly to step three, which is filter. You
6:37
aggressively clean that initial pool. You only
6:40
keep the sources that actually fit your criteria.
6:42
You force the AI to separate direct, measurable
6:45
evidence from your inference or opinion. Okay,
6:48
so we've gathered the pool and we've filtered
6:50
out the junk. But I know AI has a habit of hallucinating
6:54
or connecting dots that shouldn't be connected.
6:56
We need a way to verify its logic. That is step
6:59
four. Check. It's arguably the most crucial phase.
7:02
You interrogate what the filtered evidence actually
7:04
supports. You actively look for limitations,
7:07
biases, or uncertainty in the data. And if you
7:10
spot a glaring gap in the evidence, You pause.
7:13
You go back and search specifically to fill that
7:16
one gap. And finally, once the logic is solid,
7:19
we can actually build the thing. Step five has
7:21
to be create. Yes. You turn that verified research
7:25
into your final output. You use only the established
7:28
fact -checked findings. You stay strictly and
7:31
stubbornly within what the evidence actually
7:33
supports. Let's loop back to the research stage
7:35
for a second. Why do we ask the AI to establish
7:38
an approval standard before we even start reviewing
7:40
the links? Because if you don't lock in what
7:44
approve or reject actually means beforehand,
7:46
the AI will get sloppy. It might let weak, unverified
7:50
forum posts slip into a high -stakes client briefing
7:52
just because the keywords match. You have to
7:54
define the strict threshold equality first. So
7:56
set the strict rules first so the AI doesn't
7:58
lower the bar later. That is the perfect way
8:00
to look at it. So if we are setting these strict,
8:03
non -negotiable rules, we need to know exactly
8:06
which tool is handling the heavy lifting for
8:08
us. Right, which brings us to a major comparison.
8:11
Deep research versus Gemini notebook, the autonomous
8:15
rover versus the controlled laboratory. These
8:17
are undeniably the two heavy hitters right now.
8:20
Yeah. But they serve very different strategic
8:22
purposes. Deep research is for starting completely
8:25
from scratch. You don't have any documents yet.
8:28
You just want a fast, remarkably comprehensive
8:30
report on a brand new topic. So you want to investigate
8:33
broadly and quickly. You're casting the widest
8:36
possible net into the ocean. Exactly. Gemini
8:39
Notebook, or Notebook LM, is totally fundamentally
8:42
different. It's for when you demand absolute
8:44
control over your sources. You've already caught
8:46
the fish, so to speak. You want to do ongoing
8:49
complex work with the exact same trusted evidence
8:51
base over an extended period. Like you can inspect
8:54
individual specific claims. You can compare two
8:57
distinct documents side by side. You can say,
9:01
show me where document A disagrees with document
9:03
B. But here is the crucial limitation. Notebook
9:08
LM cannot miraculously make a weak source set
9:11
strong. It only works with the exact data you
9:13
feed it. So if you feed it biased and complete
9:16
data, you will get biased and complete analysis.
9:19
Whoa. I mean, imagine scaling to a billion queries,
9:23
keeping the exact same pristine evidence base,
9:26
and just spinning out audio overviews, complex
9:28
mind maps, and data tables endlessly without
9:31
ever having to start over. It's wild. It's an
9:34
incredibly powerful paradigm shift. The final
9:36
written article is just one possible output.
9:39
Once the laboratory is stocked with good data,
9:41
you can reuse that exact same vetted evidence
9:43
over and over for different formats. But if I
9:46
accidentally upload a deeply flawed white paper
9:48
into that workspace, can NotebookLM recognize
9:50
it and fix a fundamentally flawed piece of evidence?
9:53
No, unfortunately not. It relies entirely on
9:55
the quality of the sources you selected and imported.
9:58
You'll analyze that flawed white paper brilliantly,
10:00
but the underlying flaw remains. It organizes
10:03
your evidence beautifully, but it cannot share
10:05
bad data. We're going to take a quick pause right
10:07
here, but when we come back, we're actually going
10:09
to apply this exact framework to the listening
10:13
phone mystery. Yeah, you won't want to miss how
10:15
this plays out. We'll be right back. And we're
10:17
back. So let's put this entire theoretical framework
10:20
into practice. Let's settle the massive debate
10:23
about our phones secretly recording our conversations.
10:26
This is the experiment everyone wants to understand.
10:29
We are testing the mechanism behind why highly
10:32
specific ads appear right after we talk about
10:35
something out loud. Right. We decided to write
10:38
a comprehensive tech article explaining this
10:40
phenomenon for non -technical readers, and we
10:43
chose Notebook LM as our primary workspace for
10:46
this project. We use Notebook because we wanted
10:49
more than just a one -time summary. We wanted
10:51
to continually question the evidence from different
10:54
angles. So walk us through how you applied the
10:56
workflow here. Well, step one and two were ask
10:59
and research. We designed strictly neutral prompts.
11:01
We didn't ask how do phones listen to us. We
11:04
asked the AI to gather comprehensive app security
11:07
audits, recent regulatory court cases, and raw
11:10
tech data regarding predictive advertising. We
11:13
explicitly commanded it not to assume secret
11:16
recording was actually happening. Then we had
11:18
to clean the pool. We had to filter the junk
11:20
out. Right. So before reading anything, we asked
11:23
Gemini to define a strict approval standard.
11:25
We only wanted peer -reviewed security audits
11:27
or verified technical documentation. We filtered
11:30
the links in small, manageable batches. And we
11:34
brought only the strongest, most verifiable sources
11:36
back into our closed notebook LM workspace. So
11:39
step three was, understand, we now had dozens
11:42
of highly technical sources sitting in the laboratory.
11:45
How did the AI handle that dense information?
11:46
Notebook LM was incredible here. It organized
11:49
all those dense documents into plain language
11:51
findings. It pulled out some truly fascinating,
11:54
slightly terrifying concepts. Like, for instance,
11:56
it highlighted the use of ultrasonic signals.
11:59
Okay, wait, stop right there. We need to unpack
12:00
that. How do ultrasonic signals actually work
12:03
in this context? It's brilliant and creepy. Let's
12:07
say a television commercial for that espresso
12:09
machine plays in your living room. Embedded in
12:12
the audio of that commercial is a high -frequency
12:15
tone. Human ears cannot physically hear it. It's
12:19
completely silent to you. But my smartphone microphone
12:22
can hear it. Yes. Your phone's microphone is
12:25
passively picking up ambient noise. It registers
12:29
that specific high frequency tone. It instantly
12:32
recognizes the audio watermark of that specific
12:35
commercial. It then logs that your phone, meaning
12:38
you, was in the room when the espresso machine
12:40
ad played. So it's not recording my voice saying
12:42
I want an espresso machine. It's just matching
12:44
my physical location to the television broadcast.
12:47
Exactly. It links your profile to the product
12:49
without ever understanding a single word you
12:52
spoke. Notebook LM also detailed cross -device
12:55
tracking, which is another massive piece of the
12:57
puzzle. How does that mechanism work? So say
12:59
you're talking about the toffee maker with your
13:01
spouse. You don't search for it on your phone.
13:04
But later that night, your spouse opens their
13:06
laptop in Google's best espresso machines. And
13:10
because we share a Wi -Fi router, the ad networks
13:13
know our devices are connected. Precisely. The
13:15
ad network pools your IP addresses. It assumes
13:18
that people living in the same house share purchasing
13:20
interests. So it starts pushing espresso machine
13:23
ads to your Instagram feed, even though your
13:25
spouse was the one who actually typed the search.
13:28
Wow. It turns out these predictive advertising
13:31
systems are just terrifyingly good at correlation.
13:34
They don't actually need the microphone to be
13:36
recording our conversations. They really don't.
13:38
Gathering audio files is incredibly expensive
13:41
and legally dangerous. Gathering location and
13:44
search data is cheap and highly effective. Notebook
13:47
LM even brought up cognitive biases, specifically
13:51
the frequency illusion. Right, that's when you
13:53
buy a red car. Yeah. And suddenly you see red
13:55
cars everywhere. Exactly. You scroll past 10
13:58
,000 ads a day. You ignore almost all of them.
14:01
But because you just talked about the coffee
14:02
maker, your brain selectively notices the coffee
14:05
maker ad. You assume it's a new ad, but it might
14:08
have been in your feed all week. So the evidence
14:11
overwhelmingly pointed away from secret listening.
14:14
It pointed toward massive data correlation. But
14:17
step four was check. we had to actively look
14:21
for flaws in the AI's logic. This is where we
14:24
inevitably catch the AI making mistakes. What
14:27
did you get wrong here? Well, we found that the
14:29
AI sometimes phrases its final conclusions much
14:31
more strongly than the underlying evidence actually
14:34
supports. It will take a very specific, limited
14:37
finding and artificially inflate it into a broad,
14:40
universal fact. Give me an example of that inflation.
14:43
Based on the documents, the AI initially wrote
14:46
a draft saying, add networks constantly use ultrasonic
14:49
signals to track your location. But when we clicked
14:51
the citation and read the original security audit,
14:53
the researchers actually said, add networks have
14:56
the capability to use ultrasonic signals, though
14:58
the practice is heavily restricted. That's a
15:00
massive difference. Constantly use implies it's
15:03
happening every second. Have a capability implies
15:06
it's technically possible, but... Maybe rare.
15:09
Exactly. So we forced the AI to state the strongest
15:12
neutral conclusion. We separated the direct evidence
15:14
from its speculative inferences. We made it dial
15:17
back its confidence. And finally, we hit step
15:20
five, create. And because the evidence was now
15:23
rock solid. the final output was easy to generate.
15:27
It was. It outputted a fantastic, readable article.
15:30
It used direct, clickable hyperlinks within the
15:33
text instead of clunky academic citation numbers.
15:36
It was incredibly clean and well -sourced. Wait,
15:39
I need to push back on this. Even with this rigorous,
15:42
heavily controlled notebook setup, you're saying
15:43
the AI can still overstate a finding or connect
15:46
two dots too aggressively. We can't fully trust
15:49
the final synthesis, even in the closed lab.
15:51
You absolutely cannot just trust it blindly.
15:54
Even with perfectly curated high quality sources,
15:57
the AI might misread a complex passage. It might
16:00
completely ignore a crucial contextual limitation.
16:02
It's essentially a very eager intern. It wants
16:05
to give you a definitive answer, even when the
16:07
data is ambiguous. So what is the most vital
16:10
non -negotiable action? we must take during the
16:13
check stage. You have to manually verify that
16:15
the AI's sweeping conclusions haven't stretched
16:18
beyond what the original public evidence actually
16:20
proves. You must read the connected claims yourself.
16:23
You must rein in the AI when it gets overly confident
16:26
about weak facts. Exactly. And that tendency
16:29
for the AI to get overconfident and stretch the
16:32
truth, that brings us to the ultimate unavoidable
16:35
imitation of these tools. We've talked about
16:37
how powerful they are, but where do they completely
16:39
fall flat? AI can handle a truly surprising amount
16:43
of tedious research, but there are three core
16:46
things AI simply cannot do for you. What's the
16:49
first major limitation? First, AI fundamentally
16:52
doesn't know the good enough standard for your
16:54
specific real -world job. It has no concept of
16:57
stakes. Right. If I'm writing a casual explainer
17:00
for a blog, I need a very different standard
17:02
of proof than if I'm making a multi -million
17:04
dollar business decision. Exactly. A Wikipedia
17:08
citation might be perfectly fine for a casual
17:10
Friday newsletter. That exact same citation would
17:13
be terribly irresponsible for a medical recommendation.
17:16
The AI cannot weigh those stakes. You have to
17:19
define that threshold manually. What is the second
17:22
limitation? Second, AI constantly overstates
17:25
proof. Citations strictly trace evidence back
17:28
to a source, but they don't guarantee the accuracy
17:31
of that source. Just because a beautifully written
17:33
sentence has a tiny little number next to it
17:35
does not magically make it true. It just tells
17:38
you where the AI got the idea. It might have
17:40
misunderstood the idea completely, or the original
17:42
idea might have been flawed from the start. The
17:45
citation is a breadcrumb trail, not a seal of
17:48
approval. And third, AI genuinely doesn't know
17:51
what matters or when to stop researching. It
17:53
will happily surface 10 fascinating, deeply researched
17:56
findings. But only you, with your human context,
18:00
know which of those 10 findings actually matter
18:01
to your audience. It sounds like the AI is an
18:04
incredibly eager, tireless research assistant.
18:07
It can dig up an absolute mountain of dirt in
18:10
seconds. But you are the master gold appraiser
18:13
who actually determines what shines. You have
18:15
to sift the gold from the mud. That is beautifully
18:18
stated. The AI will dig forever if you let it.
18:21
I usually tell it to stop when the mo - important
18:23
structural questions are answered. I stop when
18:26
any new research just endlessly repeats what
18:28
I already securely know. If AI is doing all the
18:31
heavy digging and summarizing, what is our fundamental
18:34
irreplaceable job at the end of the day? You
18:36
own the standard of quality, you choose the relevant
18:39
claims that matter, and you make the absolute
18:41
final judgment the AI is just the powerful engine.
18:44
You are still steering the car. AI does the heavy
18:47
lifting. but human judgment remains the absolute
18:50
final word. It has to be. Let's recap the big
18:52
idea we've covered today. Yeah, so the core philosophy
18:54
of this deep dive is surprisingly simple. The
18:57
goal of using these tools isn't to achieve perfect,
19:00
flawless research. The goal is to achieve research
19:03
that is undeniably strong enough for the specific
19:06
job you need to do right now. You have to deliberately
19:09
match the tool to the task. Exactly. Let's run
19:12
through them one last time. Use standard search
19:15
for viewing the raw, unfiltered landscape. Use
19:18
Gemini for quick conversational explanations.
19:21
Use AI mode for broad web exploration where you
19:24
still need to see the links. Use deep research
19:27
for handing off massive, complex investigations
19:29
from scratch. And use Notebook LM for ongoing,
19:33
rigorous control over your specific sources.
19:35
And you must always apply the five -step workflow
19:37
when the real -world stakes are high. Ask. Research.
19:41
Filter. Check. Create. That systematic approach
19:44
is exactly how you turn a vague, paranoid question
19:47
about listening phones into a highly reliable,
19:50
heavily vetted technical briefing. Beat. As we
19:53
wrap up, I want to leave you with something to
19:55
chew on. If AI is eventually doing 99 % of the
19:58
gathering, the reading, and the synthesizing,
20:00
does our human ability to establish a strict
20:02
approval standard become our only remaining intellectual
20:05
currency? Beat. The next time you ask an AI a
20:08
question, pause and ask yourself, who is actually
20:10
deciding what is true? Two sec silence. Thank
20:13
you for joining us on this deep dive. We'll catch
20:14
you next time.