Latest / Never Post / What Happens When Search Goes Full AI

Transcript

What Happens When Search Goes Full AI · Never Post

0:10
Mike Rugnetta: Friends, hello, and welcome to Never Post, a
0:14
podcast for and about the Internet. I'm your host Mike
0:16
Rugnetta. This intro was written on Tuesday, 06/23/2026 at
0:21
10:35AM eastern, and we have a fantasmagoric show for you this
0:28
week. In this episode, Audrey chats with assistant professor
0:31
at the University of Maryland College of Information, Mauls
0:34
Souder, and writer, researcher, and community organizer, Erin
0:38
Kissane, about the incursion of AI models hallucinations into
0:43
increasingly prevalent corners of media, law, government, and
0:47
culture, and what this means overall for the production of
0:51
knowledge. But first, we're gonna take a quick break.
0:55
You're gonna listen to some ads unless you're on the member
0:57
feed. And when we return, we're gonna talk about a few of the
1:00
things that have happened since the last time you heard from us.
1:04
Zephyr in the sky at night, I wonder. Do my tears of morning
1:08
sink beneath the sun? She's got herself a universe gone quickly
1:12
for the call of thunder threatens.
1:15
Six stories this week. France says to Palantir. French PM
1:25
Sebastian Lecourneau announced last week that French
1:27
intelligence agency, the general directorate for internal
1:31
security, would be cutting ties with the techno fascism purveyor
1:34
in order to reduce reliance on American technology. Lecourneau
1:40
says the gap will in time be filled by French AI developer,
1:43
CHAP's vision. This comes as England's NHS, the National
1:47
Health Service, considers ditching its Palantir contract
1:51
under pressure from MPs and workers alike over privacy and
1:55
security concerns.
1:56
Now I don't wanna speak too soon here, but I'm starting to think
2:00
that maybe Palantir is bad.
2:08
Mols Sauter: No.
2:09
Mike Rugnetta: China has been increasing its global telecom
2:12
presence in South America. The most recent partnership offering
2:15
being made to Chile. State owned China Mobile offered to
2:20
construct an undersea cable linking the two nations. But in
2:23
February of this year, rest of world reports, The US stepped in
2:27
and sanctioned the South American nation, revoking the
2:30
visas of three officials. This after a similar 2019 scheme was
2:35
kiboshed also due to US pressure.
2:37
The state department has said the infrastructure projects
2:40
would have, quote, compromised critical telecommunications
2:43
infrastructure and undermined regional security in our
2:46
hemisphere, end quote. The US and China remain engaged in an
2:50
ongoing data center and AI development arms race that
2:53
pretty much no one has asked for or benefits from. We. Music, all
2:59
of it has been ingested by AI. Writing for the Atlantic, Alex
3:03
Reisner reveals that he was able to get a hold of four training
3:06
datasets to give a sense of their size.
3:09
Only one is comprised of 12,000,000 tracks that would
3:13
take 91 to listen through. AI developers and researchers
3:18
employ these datasets to develop their products, though some,
3:21
like Suno, say there are guardrails in the technology to
3:24
prevent against the models generating faithful or near
3:28
copies of existing music. Reisner writes quote, what the
3:32
datasets illustrate primarily is the scale and variety of music
3:36
easily available to AI developers. Companies often
3:39
claim to use only content that is freely available online, but
3:44
the datasets reveal the quantity of downloadable music that
3:47
developers can access even though it is not supposed to be
3:50
free, end quote. The dataset is searchable if you wanna go see
3:54
if your work has been gobbled up by the endless ravenous maw of
3:57
Silicon Valley.
3:59
YouTube has settled a Florida case in which a teenager named r
4:04
k c in court documents alleges the platform was designed to be
4:08
addictive leading to anxiety, sleep deprivation, and other ill
4:12
effects. R k c is also suing Meta, TikTok, and Snap in
4:16
California according to the BBC. This comes after a string of
4:20
various settlements and findings in users favor regarding the
4:24
deleterious effects of features like infinite scroll,
4:27
algorithmic recommendation, and so on. Most notably, six months
4:31
ago, Australia banned teens from using social media entirely
4:37
countrywide, a massive undertaking which researchers in
4:41
the British Medical Journal this month note has been particularly
4:45
ineffective writing, quote, we found insufficient evidence to
4:50
conclude that exposure to the act had any early substantial
4:53
effects on social media use among adolescents aged 16 years.
4:59
The findings contrast with similar legislative requirements
5:03
of web based services.
5:04
For example, in The UK where access to pornographic websites
5:08
substantially reduced by 30 to 50% following the introduction
5:13
of age assurance measures, end quote. Whoops. Oh, well, I
5:20
guess. Oops. Mister Rogers is on YouTube.
5:25
Okay. Maybe it's not so bad out there after all. The channel
5:27
will feature clips as well as full episodes many of which have
5:30
not aired or been otherwise available for quite some time.
5:34
The Fred Rogers Foundation has partnered with British digital
5:36
content agency, Little Dot Studios, which as of 2020
5:41
managed over 600 branded social video channels for various
5:47
clients. Won't you please?
5:50
Won't you please? Please won't you be my neighbor? My neighbor?
5:59
I'm glad we're together again. And finally, are you seeing what
6:04
I'm seeing?
6:05
Hallucinations in the news. At their recent IO conference,
6:09
Google unveiled their redesigned search product in which results
6:12
are delivered not as a list of links, but a chat with an LLM.
6:16
Searchers have found this experience increasingly
6:19
prevalent as the company has rolled out its AI overviews.
6:23
Summaries of search results aimed at synthesizing available
6:26
information without requiring additional clicks. But there's
6:30
one problem.
6:31
Well, there are actually many problems, but one is the well
6:35
documented prevalence of overview hallucinations, which
6:40
recently a German court has ruled Google is liable for. The
6:45
ruling describes the overviews as quote, the defendant's own
6:49
statements. It alone has influence over the AI's offering
6:53
and the algorithms with which the AI operates, end quote.
6:57
Google's got good company in its hallucinatory woes. Number
7:00
Cruncher's KPMG had to recently retract a report titled of all
7:04
things redefining excellence in the age of AI because it was
7:08
riddled with errors and inventions.
7:12
Ernst and Young recently retracted a report on loyalty
7:15
programs because it contained hallucinations. Wikipedia has
7:19
unsurprisingly had to ban AI contributions because they are
7:23
often factually incorrect. And it feels like every week there's
7:26
a new story about a law firm caught using AI after citing
7:29
some nonexistent precedent. What is going on? How do we
7:36
understand and protect ourselves against the incursion of opaque
7:41
machine invention into what have otherwise been the most
7:44
calculatory, document dependent, and knowledge based industries.
7:50
And what if these hallucinations are not a bug, but a feature? A
7:54
fundamental aspect of the technology revealing in some way
7:59
what it really is.
8:00
Audrey Evans: Uh-huh.
8:01
Mike Rugnetta: We're gonna take a quick break, and when we
8:03
return, Audrey talks with Malz Sauter, assistant professor at
8:07
the University of Maryland, where they teach the history and
8:10
philosophy of technology, and Aaron Kissane, a writer and
8:14
researcher focused on knowledge and networks at unbreaking.org
8:18
and previously, the COVID tracking project. They'll all
8:21
talk about what AI hallucinations actually are from
8:25
a knowledge production standpoint and how the
8:28
information they contain reflects a much longer and
8:31
stranger facet of human culture than you may have previously
8:35
considered.
8:48
Audrey Evans: In May, Google announced that they are turning
8:51
from legacy search, the search box, to AI search as the default
8:55
search mode. A May 16 TechCrunch article even had the headline,
9:00
Google search as you know it, it's over. The ground of the web
9:05
is shifting. The very way we step into our interactions with
9:09
the web is changing quickly. Google suggested this as a quote
9:13
paradigm shift, moving users from search into generative UI
9:20
experiences.
9:22
So instead of returning links, Google search will now be
9:25
dropping users into quote AI powered interactive experiences
9:30
and be dispatching information agents to gather information on
9:34
a user's behalf. So search will now look more like a long back
9:38
and forth conversation where the user will also be suggested
9:43
prompts, directions. This is really dropping users into an
9:49
entirely new research process experience. One even further
9:53
intermediated by Google and driven by the incentives and
9:56
priorities of this company. This paradigm shift that they're
10:00
talking about is one that's now less about information retrieval
10:03
as a process and now something that's geared towards prompting
10:08
users to quote, take actions and actions that are going to be
10:13
defined by their models.
10:16
I know both of you study the way we collectively use and
10:20
synthesize information, how we turn information to action,
10:24
sometimes collective action in our society. So I wanted to get
10:29
your reactions. And Erin, it is such a delight to be here with
10:33
you today. Mols, let's start with you.
10:40
Mols Sauter: This is a really interesting question because
10:42
there are a bunch of different things that are going on both
10:45
inside sort of what we'll call sort of like the AI this current
10:49
AI moment, but also these different specific systems. So,
10:53
like, let's just take the search bar. And I love that you
10:58
specifically flagged the paradigm shift. Whenever someone
11:01
says paradigm shift, I'm like, cool. Now we have to talk about
11:04
what a paradigm shift actually is and what it what it means to
11:07
have a paradigm shift.
11:09
And so what we can think about it as when did search stop being
11:14
we get returned results that are pointers to things that exist,
11:19
that people have written and they've put on the Internet as
11:24
objects, and we find them via object identifiers. Two, we get
11:30
a natural language response from a technical system that also
11:35
points to sometimes these objects. It has some
11:39
relationship to these objects, but it is not the object. It's
11:42
not merely a referent. The point of the search bar is not to
11:44
direct.
11:45
The point of the search bar is to answer. So how do we get this
11:48
paradigm shift? And one of the reasons I think that it doesn't
11:51
feel like a paradigm shift to us, the reason why, like, for
11:54
the users, we're like, wait a minute. We're not advancing.
11:58
This isn't an advancement.
11:59
This isn't better. I'm not happy about what is happening right
12:02
now. I don't feel like my experience has become more, is
12:06
it's not actually following what causes paradigm shifts. So one
12:09
of the key things in a paradigm shift in scientific theory is
12:12
the crisis and anomalies stage, which is when you've got the
12:17
theory, it is not paradigmatic yet, you are collecting your
12:21
experimental results, you're building your literature, you're
12:23
building your advocates, you're you're assembling your team, and
12:28
then you enter into that period of norm what we call normal
12:30
science, which is when we've accepted the paradigm. Now we're
12:32
just sort of like cleaning up around the edges.
12:34
We're puzzle solving. Maybe it's an even entering into an
12:37
engineering mode where we're just sort of like, yeah, this is
12:39
a thing that solves problems. We use it to solve problems. We're
12:42
not necessarily discovering anything new in the normal
12:44
science mode. But then we enter into crisis and anomaly stage
12:48
when things stop working.
12:49
We start building up these little inconsistencies, these
12:52
little holes, and they start accumulating to the point where
12:55
the the theory or the paradigm can no longer sustain itself,
12:58
and it enters into a period of crisis. And that's when things
13:02
open up for a new paradigm. That's basically the structure
13:05
of, like, how we how do we change these big ideas? And one
13:09
of the things that is currently happening is the old ideas have
13:13
never arrived at the crisis and anomaly stage. The idea that a
13:17
search bar points you to resources was perfectly happy
13:21
chilling out in the puzzle solving, like like, it's not
13:25
science, but quote unquote normal science space where it's
13:27
like, this is good.
13:28
This works. We're happy. We don't have lots of problems.
13:31
Like, there are some problems, but they're not fatal problems.
13:34
And they're certainly not fatal problems with the basic
13:35
mechanism.
13:37
Instead, we have sort of this forced paradigm shift where a
13:40
bunch of companies, and by a bunch, mean, like, four, are
13:44
pushing an entirely new way of both interacting with the web,
13:48
interacting with certain types of knowledge, and interacting
13:52
with certain types of media. And everyone's reaction is quite
13:56
expected where it's like, wait a minute. Wait a minute. We
13:59
weren't ready. Like, we're not ready to do this.
14:01
There was nothing wrong with that in a fatal way.
14:04
Erin Kissane: I think it's it's impossible to overemphasize how
14:08
hard some of these tech companies have been trying to
14:10
make paradigm shift happen for years now. So, obviously, this
14:15
is in continuity with trying to make metaverse happen, trying to
14:19
make NFTs happen, trying to make crypto happen in specific ways.
14:24
Not that any of those technologies did not exist, but
14:28
none of them existed in the ways that the tech companies pushing
14:31
them were presenting them to us. So especially given that a lot
14:36
of the folks leading these technology companies were around
14:39
for what I would classify as a genuine paradigm shift, the
14:44
arrival of search, the arrival of what used to be the open web
14:48
as a pretty substantial shift in the way that we interact with
14:53
knowledge as societies. That actually happened and they
14:58
figured out gradually how to enclose more and more of it, but
15:03
then they kind of hit a wall.
15:05
I don't want to be fair to anyone, but to be fair, I will
15:09
say the arms race of search has led to, I think many of us
15:14
think, a degraded experience over the past maybe five to
15:19
seven years sharpening more as we approach the present. And
15:23
when I say arms raised, I mean, it's the the old school pre AI
15:27
content farms and the way that the poor recipe bloggers just
15:31
started adding more and more stories about their great
15:34
grandmothers to the beginnings of all their recipe posts to try
15:37
to get the attention. There was a lot of really peculiar sort of
15:41
perverse incentive minuet happening that made search
15:47
harder and harder to deal with. If you were really good at
15:49
search, you could still figure it out. But for, I think, casual
15:53
use outside of any given person's area of expertise, it
15:56
did get frustrating.
15:58
But especially, Audrey, I appreciate you talking about the
16:01
idea of taking action, moving everyone into increasingly
16:06
enclosed knowledge spaces with prompts. It feels to me like
16:10
we've rebuilt AOL, but it talks on one hand and that we're
16:16
putting people into society wide wizards on the other, neither of
16:21
which really constructs us as subjects with agency. So,
16:25
obviously, I have, like, deep rooted philosophical problems
16:29
with this. But I do think even though many of us might approach
16:33
our refrigerators and say, I have two heads of wilted kale
16:38
and three eggs. What am I having for dinner?
16:41
There is a pretty strong sense among at least the people that
16:46
that I know who are not in this nerd zone that that is
16:50
qualitatively a different thing than what we do when we try to,
16:54
for instance, educate our children or learn about the
16:57
world or do research or do scholarship. So there's a pretty
17:02
strong pushback in some aspects, but it bleeds into this. Yes,
17:06
but there's the convenience factor with a class of things
17:10
that perhaps can be answered by a machine and we're perfectly
17:14
happy with that. Having those two modes, which in our own
17:17
lives shift in and out based on how tired we are, what we're
17:21
trying to accomplish, where we are and we're trying to
17:23
accomplish it complicates this a lot and backed a lot of people
17:27
into positions where they're doing things that they perhaps
17:30
wish they were not doing.
17:33
Audrey Evans: So those complications Erin brings up
17:35
remind me of ideas I've heard you bring up before malls around
17:39
ground truthing. Would you take us through some of that?
17:42
Mols Sauter: One of the things I find interesting about LLMs at a
17:45
theory level and sort of at a philosophy of technology level
17:48
is the ways in which they counter expectations that we
17:52
have about what computers do. And that makes the reaction to
17:56
them complicated. And this sort of goes along with sort of the
18:00
ground truthing and the ground truthing problem, which is a big
18:03
problem in LLMs. Because if you think about what LLMs are and
18:07
how they're constructed, their ground truth is not our truth,
18:10
our ground truth. Our ground truth is based on the fact that
18:13
we have bodies.
18:14
They live in a physical environment. We have sensory
18:18
access to that environment. But we expect a lot of things that
18:22
we talk about in human communication, these bits of
18:26
bits of sort of phonetic sound and text that we throw at each
18:29
other, we expect those things to be related to that environment.
18:33
That's our ground truth. We're like, this is these things have
18:35
always coincided and when they don't coincide, that's bad.
18:39
That's a lie. That's a falsehood. That's a mistake.
18:42
It's not what we want. Computers have a very long history of
18:47
being portrayed in sort of both broader culture, in politics, in
18:51
science, and technology itself.
18:52
It's like, this thing is tied to a set of ground truths that we
18:57
call mathematics. And mathematics, it has a certain
19:00
epistemic status. And we also expect it to progress through a
19:04
series of logical steps. This is how it functions. We're going to
19:07
walk through these steps.
19:08
This is how a computer program functions. And the listeners
19:10
cannot hear me put many scare quotes on many of those phrases.
19:14
But that's sort of the general understanding of how computation
19:17
works. But these systems do not exist as ground truths, we live
19:25
in the outside world and we must refer to it in our speech, text,
19:28
and action, or as pure mathematical systems of this is
19:33
a logical progression through this set of mathematical steps
19:35
and this lives in this epistemic frame. These systems exist in a
19:41
big sea of language and linguistical associations, and
19:45
that is an unground truthed situation.
19:49
But internal to the LLM, the logic is not I need to produce
19:53
things that are related to an external ground truth. The logic
19:55
of the LLM is I need to present the most acceptable string of
20:00
tokens based on strings of tokens that I have. The thing
20:04
that is done is a performance. And it's a performance within
20:09
the bounds of genre and how genre is constructed and
20:14
reflected in these associative groupings of tokens. And this is
20:19
a real problem for how we think about computers because we're
20:23
used to coming to a computer and being like, math problem and
20:25
having computer be like, math problem.
20:27
Answer to math problem. Here you go. And I think this is why
20:30
people reacted so aggressively or like pinned very strongly on
20:36
it does math wrong Because that's the basic thing you
20:39
expect a computer to do is you expect it to math right. The
20:42
whole purpose of computers is math right. These do not math
20:46
right.
20:46
These other things write and they they other things write in
20:50
a weird way. And I think that the weird way they do it is
20:53
really relevant to, like, to how we think about those other
20:56
things. But because they're in a computer and they present as a
21:00
computer, it's causing, like, this huge collapse.
21:05
Audrey Evans: So this reminds me of how this all started, Erin,
21:08
with your posts on Blue Sky exploring how we orient
21:12
ourselves and maybe even what postures might be most useful to
21:16
take when we interact and engage with these systems.
21:20
Erin Kissane: People talk about anthropomorphization. I don't
21:22
think that's quite what's happening here. But when we see
21:25
the computers stop mapping and we see them talking and
21:30
especially when you said, Audrey, moving toward a back and
21:34
forth for search, I think the back and forth is central to the
21:40
weirdness of our moment in LLMs because we are experiencing
21:45
them. Many of us, most of our population who touches this
21:49
stuff is probably doing it in a conversational way and you only
21:52
have conversations with entities. The entities are
21:56
obviously not people.
21:59
I don't want to get too bogged down in entity ness, but because
22:05
of my particular background, which comes out of religion and
22:09
folklore, this sure looks familiar to me. We actually have
22:14
centuries of
22:17
Mols Sauter: We have lots of this.
22:18
Erin Kissane: Of cultural technology about how you deal
22:23
with weird entities, and this predates psychology. We have a
22:29
pretty strong tradition of things that are perhaps demons
22:33
or that are perhaps gods or angels or spirits or creatures,
22:38
things that provide us with divinatory knowledge. And in the
22:42
same way that Miles was saying, you know, like, we expect
22:45
computers to first be math, We kind of expect weird nonhuman
22:49
entities that will talk back to us and give us advice to be a
22:55
particular kind of thing. I think it is less like talking to
22:59
a hallucinating entity and more like participating in
23:03
hallucination. When you are in one of these weird zones, you
23:07
know that you are not getting ground truth.
23:09
You know you are someplace weird. And specifically, I think
23:12
there's something that comes out of, like, cross cultural
23:15
something sort of below fairy tales and into like spirit
23:19
stories, folk stories, ghost stories, this stuff. The idea
23:23
that the entities have a coherent system, but that you
23:26
can't quite understand it. So your best bet is to sort out
23:30
what kind of contract you can make with this class of entity
23:34
or with animals or with other aspects of the natural world so
23:38
that you have a crossing over point. None of these
23:40
technologies allow for or permit just a continuous flow of back
23:46
and forth with any of these entities or systems, I think
23:50
that is instructive or it could be instructive for us about our
23:55
own weaknesses to a continuous back and forth with something
24:00
that's not human, that's not practicing according to our
24:03
rules, that is not coming from embodied knowledge, that has
24:06
either no ground truth or a ground truth that we can't
24:09
really understand.
24:10
Audrey Evans: I love this and I haven't heard anyone else
24:13
articulate this quite like you do. I feel like you're inviting
24:17
us into a more honest understanding of the
24:22
relationality that's happening and inviting us to remember that
24:27
the agency and where we define the relationship to this
24:31
technology and how we interact with it has to start in our
24:35
bodies and end in our bodies.
24:38
Erin Kissane: And and I don't wanna take away from the broad
24:41
political and anti labor aspect of this technology, but I do
24:46
like it it frustrates me that the relational part, that
24:51
conversation has been pretty crude. Yeah. And I think people
24:55
are perhaps embarrassed to talk about it unless they somehow
24:58
lack embarrassment and are happy to say, here are my 300 pages of
25:02
chat log and why I think my Claudine is special.
25:06
Mols Sauter: Just jumping off of the last point, the actual skill
25:10
set you need first, I was like, you absolutely 100% need to have
25:13
a very core understanding of the fae. This is this thing that is
25:17
not telling me things in a way that I understand things to be
25:20
telling me things. And you have to sort of be able to hold that
25:23
in your head if you're gonna have an extended interaction
25:25
with it. When I started looking at it and I started looking at
25:28
it, it is like, what can it do? What is the thing that it is
25:31
capable of doing?
25:33
One of the first things that popped up to me was like, oh,
25:35
what you need to make this work is discernment. I know the shape
25:39
of what you should be making. You have made things in that
25:41
shape, but something around this is is off. Something around this
25:44
is not functioning. Something around this is not what I
25:47
expected.
25:48
Let's go figure out why this isn't this way. And I think most
25:53
people use it like, people, especially with the way they're
25:55
being distributed now, people engage with them in a one off.
25:59
They're like, I have typed something into a search bar. I
26:00
have gotten a result. I need to check something.
26:03
Students will ask it to summarize things. One of the
26:05
major projects we're working on this summer is about summary and
26:08
how summary functions with that within different models. And
26:11
also what a broader theory of summary is, which it turns out
26:14
is quite difficult to pin down. I'm gonna bring in Lacan because
26:19
I'm a theory sicko. Lacan has a concept called the subject
26:23
supposed to know.
26:25
And the subject supposed to know is the position occupied by the
26:29
analyst in a psychoanalytic relationship. And the analysand
26:33
enters into the analytical relationship and projects upon
26:37
the analyst the position of the subject supposed to know. They
26:41
are the person who so the the analysand is coming in and
26:44
saying, I am full of weird shit. I am full of nonsense. I'm full
26:48
of contradictory things.
26:49
I'm full of compulsions. I'm full of, like, blockages. I'm
26:52
full of all of this garbage, and I can't figure it out. That guy
26:57
can. That's he's the subject supposed to know.
26:59
He's gonna know. He he's gonna take all of my utterances and
27:03
he's going to put them in order and he's gonna make it make
27:05
sense to me. And the subject's most to know position is a
27:09
collision of a bunch of things. It's a collision of what this
27:13
what the analysis and previous experiences are with people,
27:16
with people older than them, with therapists, with doctors,
27:19
with teachers, whatever. Their knowledge about psychoanalysis,
27:24
their knowledge their the environment that it is happening
27:26
in, who the analyst presents as, how they've talked about
27:31
themselves.
27:32
All of this shit comes down into the subject supposed to know
27:35
position. And importantly, the analyst doesn't know. That is
27:39
their function is not to know you better than you know
27:41
yourself. Their function is to sort of create a discursive
27:44
construct where you can explain yourself to yourself and
27:47
eventually do it in a way that makes sense to you. And I would
27:52
like to posit that we approach computers in general, but
28:00
specifically discursive computers and place them into
28:04
the subject supposed to know position.
28:06
I like this theory and it's very effective for me because it
28:08
explains a bunch of different types of interactions. It
28:10
explains the one off interactions. It explains the
28:14
more the multi turn interactions. It explains why
28:17
people think that LLMs can stand in for constituent data or all
28:22
these other things because it is the subject supposed to know.
28:24
And like the projection of the subject supposed to know, it is
28:27
a it is a collapse of a bunch of stuff, including, like, the vast
28:31
history of computing as computers, math, and math knows.
28:35
And so when so when we enter into these interlocutor
28:40
positions where what we are experiencing is a one to one
28:45
interaction. But what is actually happening is a one,
28:50
that's me, to one, the interface of the LLM, to large entity,
28:57
meaning the training data and all of the training shit that's
28:59
happened before to everyone. The guy you're talking to is the guy
29:04
talking to everybody. Like, even though it seems like you it's
29:07
not just talking to you, it's talking to everybody. Everyone's
29:09
getting those weird weather metaphors.
29:11
Everyone's getting delve. Everyone's getting these words
29:14
over and over again. It occupies these like, it's stepping into
29:18
these existing social and cultural constructs and
29:24
animating them in a way that is deeply destabilizing. So I think
29:28
Erin is totally right that we have all of these stories. We
29:31
have all these ways of understanding these things.
29:34
Erin Kissane: Thinking about subjects supposed to know in
29:36
this sort of post therapeutic turn of society, again, I get
29:43
pulled back into who we asked to do that kind of organizing work
29:49
for us before that turn, were frequently whatever kind of
29:56
spiritual advisors we depended on. I think that work that Malz
30:03
is doing to break down our own positioning, our expectations of
30:09
the role, really important and could be really useful in
30:14
helping people pull out of the mode that you might use if you
30:19
bring these kinds of problems to someone who is either an
30:23
explicitly spiritual authority. I am bringing it to my priest. I
30:27
am bringing it to my minister. I'm bringing it to someone in my
30:30
zone who has ethical authority if they're not religious, to an
30:35
elder.
30:36
Those people are intended to give you different kinds of
30:39
answers. Those people are intended to give you the
30:43
position of cultural wisdom and authority and advice that will
30:49
help you find your way out of whatever quandary you're in.
30:56
That's a really dangerous posture to bring to these
31:00
machine systems. So I think that kind of picking apart is
31:06
essential work and I'm excited that it's happening. I also
31:09
snagged a little bit on discernment because it's such a
31:13
it's it's both such a great concept.
31:15
And I don't know if this will be clear to those of us who didn't
31:18
grow up as church kids, but the age of discernment is a big deal
31:25
in at least the forms of Protestant Christianity that I
31:29
grew up in. I think there are very similar structures across
31:34
at least a bunch of the Christian tradition. But there's
31:38
the idea that until children hit the age of discernment, you need
31:44
to treat them as a slightly different kind of creature. They
31:47
don't have moral reasoning. They don't have the lived experience
31:53
to confront really difficult questions.
31:57
Where we reconstruct discernment for this moment, what
32:03
discernment is going to be for the children, teenagers,
32:08
students, all of us approaching these weird systems feels like a
32:14
really live question to me.
32:16
Mols Sauter: I think can I can I just give a brief defense of,
32:18
like, a thing that I say a lot people that frequently yell at
32:20
me for, but I'm just like, no, this is a very this is how I
32:23
think about it and I refuse to stop? Yeah. Okay. So Susan
32:26
Sontag in On Photography. At the end of On Photography, she's
32:29
presenter at the end of the essay that that is the the money
32:32
essay in On Photography.
32:33
She says, okay. There are now both there's the ecology of
32:38
reality and the ecology of images. And over the entire part
32:41
over this entire essay, she has argued, the ecology of images is
32:44
basically taking over and re and, like, re accounting for the
32:49
ecology of reality. And this is a problem. She does not like
32:52
this.
32:53
And I think on photography is one of the most important essays
32:56
you can read about data and data systems, not just computer
33:00
vision systems, but any data systems. Instead of having an
33:04
ecology that I can access and look at and contest, you have
33:09
created a self referential system that only relies on
33:13
itself to define its own reality. And that is what
33:16
psychosis is. That is when you have a self referential reality
33:20
that is not available for intervention. When people
33:24
attempt to intervene, that input is taken and translated into the
33:28
self referential system and is not permitted to destabilize it.
33:32
And that's what this is. That's these are like, when you have
33:36
nested realities in this way, that is what is occurring. So
33:40
when I sit when we're talking about the hallucination framing,
33:43
the reason why I don't necessarily like the
33:45
hallucination framing is because, a, of all things become
33:50
hallucinations only when they are offered to someone who is
33:52
outside of the self contained system. So it could be that yes.
33:56
You're receiving something that you are defining as a
33:58
hallucination.
33:59
Internal to the system though, that is that is the reality.
34:02
That is the that is the natural production of the context that
34:05
it has. I think, like, when we're talking about systems and
34:10
interactions and ways of interacting with things that are
34:13
fundamentally occupying separate and contained realities from the
34:18
one that the we all exist in, you really can't discount that.
34:24
Erin Kissane: I think my intervention on top of that is
34:27
to loop back around to this question the Dodri brought us at
34:33
the very beginning about what happens when search stops being
34:38
a list of references and starts being a conversation in a closed
34:44
and as you just emphasized, actually quite sharply bounded
34:49
subset of material. And of course, what's in the the
34:54
training data is not everything that's on the Internet. It is a
34:57
lot of awful things that are on the Internet, but even that is
35:00
not everything and what's on the Internet is a subset of
35:03
everything. So there's all that shaping that was already going
35:06
in. And then we have these and I actually because I'm looking at
35:11
the technical definitions, I was looking into psychosis and
35:14
hallucinations next to each other because of course in human
35:17
psychology they're often related.
35:19
We are running these systems that work in this closed way
35:26
over this highly shaped dataset. Additionally, there are the
35:31
incentives that the corporations producing these systems are
35:36
applying. It becomes impossible for those of us on the search
35:41
box end to understand which of these factors we're
35:45
encountering, which mixture of these factors we encounter in
35:49
any given interaction, which is also profoundly destabilizing
35:53
and leads, I think, to a lot of folksonomy of knowledge. This is
35:59
happening because someone wanted it to happen versus this is
36:03
happening because the machines have a psychotic construct of
36:09
moving through the world. This one is happening because
36:12
training data is what it is.
36:14
It's it's impossible for us to know. Therefore, we are going to
36:18
make so much shit up about what is happening. At the same time,
36:22
we are also being, I think, substantially destabilized in
36:27
our understanding of our own ability to think and know
36:31
things. That's a pretty volatile combination when you are feeling
36:34
destabilized as someone who may have less agency or subjectivity
36:41
than you would expect and you're encountering all this stuff. It
36:45
feels like a powder keg to me culturally.
36:48
I don't think we tend to see great things happen in eras
36:53
characterized by really extensive spiritual malaise
36:58
about our ability to know things and take action. Not to end on a
37:04
bummer, but I do think these things are worth watching
37:07
closely as we bring all of this to everyone whoever touches
37:12
search or asks their phone a question.
37:14
Mols Sauter: Just a shout out quick shout out to Nick Siever
37:17
and the book and his book Computing Taste and the concept
37:19
of the Heterogloss Corporation. That is a really good sort of
37:23
set of ideas to think through like how a bunch of different
37:26
things get combined, like Yeah. Collapsed down into the one
37:29
result, the one interface, the one he's talking about a company
37:33
that is definitely maybe not Spotify, but great So good. Also
37:38
super readable.
37:43
Audrey Evans: This was my dream interview. I had so much fun.
37:48
Mols Sauter: It's the interview with the reading list.
37:49
Audrey Evans: Yeah. Thank you both so much.
37:52
Erin Kissane: Thank you, Audrey.
37:55
Audrey Evans: You can find Mauls and Aaron all over the web, but
38:01
I'm gonna lead you to where this all started on Blue Sky when I
38:05
stumbled across a really interesting conversation they
38:08
were having. You can find Mauls at oddletters dot bsky dot
38:14
social and Erin at kisane.my@proto.social.
38:38
Mike Rugnetta: That is the news we have for you this week. We're
38:53
gonna be back here in the main feed on or around Thursday, July
38:57
2. How do we love $4 a month for a Never Post membership? Let me
39:03
count the ways. One, two, three, four.
39:16
That's it, I guess. Neverpo dot s t. Become a member today.
39:21
Never Post's producers are Audrey Evans, Georgia Hampton,
39:24
and the mysterious doctor, first name, last name. Our senior
39:28
producer is Hans Buto.
39:29
Our executive producer is Jason Oberholzer. Show's host, that's
39:32
me, is Mike Rugnetta. Moving variant ornithography of those
39:39
uninitiated made into memory by the me briefly incarnate. Full
39:44
of myself on successive nights dense and alone sings you back.
39:49
Need keeps the book of dying open, the language common after
39:53
all.
39:54
Excerpt of fourth fourth from the whole note by Gil Ott. Never
39:59
Post is a production of charts and leisure and it's distributed
40:03
by Radiotopia.