Latest / Mind Cast / The Unwritten Mind | Why AI Only Learns from Our Mask

Transcript

The Unwritten Mind | Why AI Only Learns from Our Mask · Mind Cast

• 0:00
  SPEAKER_00: Imagine the world's most powerful AI, trained on essentially every book, article, research paper, and digitized web page ever produced by human civilization.
• 0:11
  SPEAKER_00: Now, ask it this: what does it actually feel like to land a plane in a crosswind?
• 0:17
  SPEAKER_00: What is the precise somatic sensation in your fingertips when the tissue you're cutting during surgery is just a few millimeters from a critical artery?
• 0:27
  SPEAKER_00: Or how do you know, not intellectually, but in your gut, which budget argument is quietly going to kill your project before it gets off the ground?
• 0:37
  SPEAKER_00: The answer, and this should genuinely unsettle you, is that the AI cannot know, not because it hasn't read enough, but because no one ever wrote those things down.
• 0:50
  SPEAKER_00: And the things that never get written down, those might be the most important things of all.
• 1:06
  SPEAKER_00: I'm your host Will, and today's episode explores one of the most underappreciated structural limitations of artificial intelligence.
• 1:15
  SPEAKER_00: By the end of this episode, I promise the way you think about AI, human expertise, and the nature of knowledge itself is going to shift in a meaningful way.
• 1:25
  SPEAKER_00: Here's the central question.
• 1:26
  SPEAKER_00: If AI systems are trained on the totality of human written text, does that mean they have access to the totality of human knowledge?
• 1:35
  SPEAKER_00: The intuitive answer, the one that most AI hype tends to assume, is yes, but there is a school of thought, grounded in philosophy, sociology, and cognitive science, that says this assumption is fundamentally wrong.
• 1:49
  SPEAKER_00: The framework we're exploring today is called the manicured knowledge paradox.
• 1:54
  SPEAKER_00: It argues that the digitized text of humanity is not a mirror of human cognition, it's a filtered artifact of performance, and large language models are learning from that performance, not from the raw, embodied reality of what it means to be a thinking human being.
• 2:12
  SPEAKER_00: Here's what we'll cover.
• 2:13
  SPEAKER_00: First, the epistemological iceberg, the vast submerged domain of tacit knowledge that text simply cannot capture.
• 2:22
  SPEAKER_00: Then, dramaturgical distortion, how all public text is, by nature, a staged performance, and third, the second-order consequences for society and institutions.
• 2:33
  SPEAKER_00: Plus, three concrete takeaways you can carry with you.
• 2:37
  SPEAKER_00: Let's go.
• 2:38
  SPEAKER_00: Key insight one, the epistemological iceberg.
• 2:43
  SPEAKER_00: Let's start at the philosophical foundation, because this is where the whole argument lives or dies.
• 2:50
  SPEAKER_00: I want to introduce you to a Hungarian British philosopher named Michael Polanyi, a scientist-turned philosopher who said something remarkably simple and profound.
• 3:01
  SPEAKER_00: We can know more than we can tell.
• 3:04
  SPEAKER_00: What Polanyi was pointing to is a fundamental divide in human knowledge.
• 3:10
  SPEAKER_00: On one side, explicit knowledge, the kind that can be written down, codified, transmitted in a textbook or manual.
• 3:18
  SPEAKER_00: It's the stuff you can Google, the stuff you can teach in a lecture hall, and critically, it's the stuff AI systems are trained on.
• 3:27
  SPEAKER_00: On the other side, tacit knowledge, somatic, it lives in the body.
• 3:32
  SPEAKER_00: It's the kind of knowing you develop through years of practice, observation, and failure, and that you often cannot put into words even if you try.
• 3:42
  SPEAKER_00: Think about a master machinist with 30 years on a lathe, they can hear, literally hear, from the acoustic vibration of the cutting tool whether metal is about to crack.
• 3:54
  SPEAKER_00: No manual tells you that.
• 3:56
  SPEAKER_00: That knowledge lives entirely in the body and ear of that individual, and when that machinist retires, a significant portion of that knowledge retires with them.
• 4:06
  SPEAKER_00: Or consider a veteran pilot managing a crosswind landing.
• 4:11
  SPEAKER_00: The adjustments they make are not the product of consulting a checklist, they're the product of thousands of hours of embodied experience and almost unconscious sensitivity to aerodynamic instability.
• 4:24
  SPEAKER_00: You can read every aerodynamics textbook ever written and not have that knowledge, because that knowledge isn't in the books.
• 4:32
  SPEAKER_00: Now, there is a parallel concept from political economy, borrowed from the ancient Greeks.
• 4:37
  SPEAKER_00: They distinguished between two kinds of practical knowledge, techne, formal written technical knowledge, and medis, practical, localized, unwritten wisdom born of experience, the cunning of Odysseus, the river crossing knowledge of a seasoned boatman, the street-level understanding of how a neighborhood works that no city planner's model captures.
• 4:59
  SPEAKER_00: AI training is overwhelmingly techne.
• 5:02
  SPEAKER_00: The Medis almost never makes it into text, because by definition it resists articulation.
• 5:09
  SPEAKER_00: This is the epistemological iceberg.
• 5:12
  SPEAKER_00: The tip above the waterline is explicit knowledge, bright, visible, accessible.
• 5:18
  SPEAKER_00: AI absorbs that at extraordinary scale.
• 5:21
  SPEAKER_00: But the iceberg's mass, the vast submerged bulk, is tacit knowledge, somatic cognition, pre-verbal intuition, situated community-embedded understanding.
• 5:33
  SPEAKER_00: Three invisible modes AI cannot access.
• 5:37
  SPEAKER_00: First, somatic and embodied cognition, the surgeon feeling tissue resistance, the machinist hearing metal stress.
• 5:45
  SPEAKER_00: None of this is in training data because none of it was ever written.
• 5:49
  SPEAKER_00: Second, preverbal heuristic synthesis.
• 5:52
  SPEAKER_00: Experienced clinicians arrive at a diagnosis before they can articulate their reasoning.
• 5:58
  SPEAKER_00: The published case report is a retrospective reconstruction, not a transcript of the actual cognitive act.
• 6:05
  SPEAKER_00: AI learns the story, not the cognition.
• 6:09
  SPEAKER_00: Third, situated socio-cultural knowledge, the unwritten rules inside communities of practice, never written down because, to practitioners, they're self-evident, invisible to AI.
• 6:22
  SPEAKER_00: Large language models do develop computational tacit knowledge, implicit semantic structures in the model's vector space.
• 6:29
  SPEAKER_00: Impressive, but it's a shadow derived from the explicit layer, without ever touching the submerged mass beneath.
• 6:37
  SPEAKER_00: The bottleneck isn't the size of the model, it's the nature of what was written down in the first place.
• 6:44
  SPEAKER_00: Key insight 2.
• 6:46
  SPEAKER_00: The dramaturgical distortion.
• 6:49
  SPEAKER_00: This brings us to the second major point, from sociology rather than philosophy, and I find it even more unsettling than the first.
• 6:58
  SPEAKER_00: In the 1950s, the sociologist Irving Goffman developed a framework called dramaturgy.
• 7:06
  SPEAKER_00: His central argument?
• 7:07
  SPEAKER_00: All social interaction is a kind of performance.
• 7:11
  SPEAKER_00: Like actors on a stage, people present different versions of themselves depending on context and audience.
• 7:18
  SPEAKER_00: Goffman distinguished between two key zones, the front stage, the formal performance, where we manage impressions and present our best, most curated selves, and the backstage, the private zone where the mask comes off, where raw motivations live, where people acknowledge mistakes, argue, improvise, and behave as actual human beings.
• 7:43
  SPEAKER_00: Now here's the critical question.
• 7:46
  SPEAKER_00: Where does publicly available text fall on that spectrum?
• 7:51
  SPEAKER_00: Almost exclusively front stage.
• 7:54
  SPEAKER_00: A scientific paper is front stage, a clean logical arc of hypothesis, method, result, conclusion.
• 8:02
  SPEAKER_00: What it almost never shows are the dead ends, failed experiments, arguments between collaborators, and grant pressures that shape the research.
• 8:11
  SPEAKER_00: All backstage, never written down.
• 8:14
  SPEAKER_00: A corporate annual report is front stage.
• 8:17
  SPEAKER_00: The backroom negotiations, political compromises, and informal power dynamics that actually drove decisions almost never in any document.
• 8:28
  SPEAKER_00: A government policy brief is front stage.
• 8:31
  SPEAKER_00: A LinkedIn post is front stage.
• 8:33
  SPEAKER_00: Virtually every form of public written communication is an act of impression management, by definition, a performance.
• 8:41
  SPEAKER_00: So when we train large language models on human written text, we are not training them on human cognition.
• 8:48
  SPEAKER_00: We are training them on the front stage of human culture, the performance, not the rehearsal, the press release, not the meeting where the crisis was actually managed.
• 8:59
  SPEAKER_00: This creates three specific dramaturgical distortions.
• 9:03
  SPEAKER_00: Distortion one, AI learns to simulate reasoning as frictionless narrative.
• 9:09
  SPEAKER_00: Because intellectual content has been cleaned up and presented as logical progressions, the model learns that thinking looks like a clean path from question to answer.
• 9:20
  SPEAKER_00: But real cognition is messy, it doubles back, it contradicts itself.
• 9:25
  SPEAKER_00: The model never learns how human breakthroughs actually happen, because that reality stayed in the backstage.
• 9:32
  SPEAKER_00: Distortion 2.
• 9:34
  SPEAKER_00: Institutional over-idealization.
• 9:37
  SPEAKER_00: Corporate and policy documents describe how decisions should be made almost never how they're actually made, through informal conversations, personal relationships, and power dynamics that never make the meeting minutes.
• 9:50
  SPEAKER_00: The model develops a structurally naive view of how institutions function.
• 9:55
  SPEAKER_00: Distortion 3, the alignment process amplifies the front stage bias.
• 10:01
  SPEAKER_00: Reinforcement learning from human feedback trains models to be polished, risk-averse, and socially appropriate, suppressing the very backstage realities that would give the model genuine depth.
• 10:13
  SPEAKER_00: Here's the critical challenge to the common assumption that bigger is better.
• 10:17
  SPEAKER_00: More text at scale gives you more breadth of the same surface, not more depth.
• 10:23
  SPEAKER_00: And with a growing proportion of AI training data now synthetic, generated by other models trained on front stage text, you're entering a closed loop, a cyclic distillation of front stage bias reinforcing its own distortions with every iteration.
• 10:40
  SPEAKER_00: Researchers have called this an epistemic echo chamber, and the chamber is getting louder.
• 10:47
  SPEAKER_00: Key insight 3.
• 10:49
  SPEAKER_00: Broader implications.
• 10:51
  SPEAKER_00: We've established the two core structural limits.
• 10:54
  SPEAKER_00: Now for the most consequential question.
• 10:57
  SPEAKER_00: What are the second-order effects of these limits on society and institutions?
• 11:03
  SPEAKER_00: Let me introduce a concept I think we'll be hearing a lot more about, algorithmic epistemic substitution.
• 11:11
  SPEAKER_00: It describes something happening right now in hospitals, law firms, engineering firms, and government agencies.
• 11:18
  SPEAKER_00: The process by which tacit human expertise, hard-won, embodied years in the making MEDIS is being displaced by the clean, fluent, structurally confident outputs of AI systems.
• 11:32
  SPEAKER_00: Here's why this is so dangerous.
• 11:35
  SPEAKER_00: The substitution often feels completely reasonable, because AI-generated text has extraordinary structural fluency.
• 11:43
  SPEAKER_00: It reads well, it's coherent, well organized, and authoritative.
• 11:48
  SPEAKER_00: And in a world where we use fluency as a proxy for competence, where we trust things that are well written and confidently stated, that authority is very persuasive, even when it is detached from operational reality.
• 12:02
  SPEAKER_00: The pattern?
• 12:03
  SPEAKER_00: An organization decides that its senior physician, veteran structural engineer, or seasoned underwriter with 20 or 30 years of accumulated judgment can be replaced by an AI decision support system, faster, cheaper, consistent.
• 12:20
  SPEAKER_00: On paper, it makes sense.
• 12:22
  SPEAKER_00: But what is actually happening?
• 12:24
  SPEAKER_00: You are swapping MEDIS for techne, replacing embodied, context-sensitive, backstage-aware expertise with a system that by definition only knows the front stage.
• 12:36
  SPEAKER_00: The consequences are what the framework calls organizational epistemic accidents.
• 12:42
  SPEAKER_00: Failures not because the AI made an error, but because its model of reality was structurally incomplete in ways that weren't visible until they mattered most.
• 12:52
  SPEAKER_00: Consider the Space Shuttle Challenger disaster, January 28, 1986.
• 12:58
  SPEAKER_00: The O-ring seals failed in cold temperatures.
• 13:02
  SPEAKER_00: The engineers had a lived, embodied sense that something was wrong, tacit knowledge, gut-level alarm bells, that the formal risk assessment process did not capture.
• 13:13
  SPEAKER_00: They raised their concerns, but the front stage institutional machinery overrode those backstage warning signals, catastrophic result.
• 13:23
  SPEAKER_00: Extend that logic to a world where we systematically replace people who carry tacit, backstage-aware expertise with AI systems constitutionally incapable of accessing it.
• 13:35
  SPEAKER_00: What happens to the warning signals?
• 13:37
  SPEAKER_00: They don't get raised.
• 13:39
  SPEAKER_00: The system doing the analysis cannot even perceive them.
• 13:43
  SPEAKER_00: The framework describes a blind spot matrix, three dimensions of knowledge AI cannot learn from public documents.
• 13:51
  SPEAKER_00: First, somatic intuition.
• 13:54
  SPEAKER_00: Remove this from decision making, and you get de-skilling and operational fragility.
• 13:59
  SPEAKER_00: Second, backstage sociopolitical dynamics, lose the human experts who understood how things actually worked, and you get institutional blindness, decisions from the front stage map, not the backstage territory.
• 14:15
  SPEAKER_00: Third, nonlinear exploratory cognition, substitute AI outputs optimized for fluency, and you get epistemic stagnation, the illusion of progress without genuine discovery.
• 14:29
  SPEAKER_00: AI excels at combinatorial interpolation, synthesizing across known domains, but it cannot infer what was never written.
• 14:39
  SPEAKER_00: The gap between a plausible sounding account and genuine operational reality is exactly where organizational epistemic accidents live.
• 14:50
  SPEAKER_00: Let me pull these threads together.
• 14:52
  SPEAKER_00: The manicured knowledge paradox is a structural argument.
• 14:57
  SPEAKER_00: The bottleneck constraining AI cognition is not compute power, not parameters, not context window length.
• 15:05
  SPEAKER_00: It's the nature of human communication itself.
• 15:09
  SPEAKER_00: Human beings do not write down most of what they know, and much of what they do write is a performance projecting an idealized version of themselves.
• 15:20
  SPEAKER_00: AI trained on this text is not learning from the mind, it's learning from the mask.
• 15:26
  SPEAKER_00: As long as the backstage remains unwritten and embodied, the limits of machine cognition are not an engineering problem, they're a philosophical one.
• 15:37
  SPEAKER_00: Three concrete takeaways.
• 15:39
  SPEAKER_00: Takeaway one, ask the second question.
• 15:43
  SPEAKER_00: Whenever you use AI to reason through something, a medical question, a business decision, an engineering challenge, ask, what tacit knowledge would a human expert bring that isn't in any document?
• 15:56
  SPEAKER_00: Who carries that embodied backstage-aware understanding?
• 16:00
  SPEAKER_00: And am I consulting them alongside the AI or instead of them?
• 16:05
  SPEAKER_00: The AI gives you the techne, make sure you're also getting the Midas.
• 16:11
  SPEAKER_00: Takeaway 2.
• 16:12
  SPEAKER_00: Treat fluency as a warning sign, not a signal of depth.
• 16:17
  SPEAKER_00: The more authoritative an AI output sounds, the more carefully you should scrutinize it.
• 16:23
  SPEAKER_00: The model produces fluent text regardless of whether the underlying understanding is operationally complete.
• 16:30
  SPEAKER_00: Ask, does this sound good or is it actually right in the ways that matter?
• 16:36
  SPEAKER_00: Takeaway 3.
• 16:38
  SPEAKER_00: Protect the environments where tacit knowledge is transmitted.
• 16:42
  SPEAKER_00: Midas is passed through apprenticeship, proximity, observation, failure, mentorship, residency programs, craft apprenticeships, communities of practice, these are under economic pressure, but if we allow them to erode, that tacit knowledge does not migrate into AI, it disappears, and the next generation will lack not just skills, but a whole category of knowing our tools cannot replace.
• 17:11
  SPEAKER_00: The question isn't whether AI is powerful, it is.
• 17:15
  SPEAKER_00: The question is, powerful at what?
• 17:18
  SPEAKER_00: And at what cost to the kinds of knowing it cannot touch?
• 17:23
  SPEAKER_00: Let's bring this home.
• 17:25
  SPEAKER_00: Today, we explored the manicured knowledge paradox.
• 17:29
  SPEAKER_00: Humanity's published text is not a mirror of human cognition, but a filtered artifact shaped by two powerful forces, the epistemological iceberg, where the vast bulk of tacit embodied knowledge lives below the surface of what anyone ever wrote down, and dramaturgical distortion, where the text that does exist is a front stage performance that conceals the messy, nonlinear backstage reality of how human beings actually think and act.
• 18:00
  SPEAKER_00: We looked at the downstream consequences, algorithmic epistemic substitution, the blind spot matrix, and organizational epistemic accidents, failures that emerge not from AI making mistakes, but from AI doing exactly what it's designed to do in contexts where what it cannot know is precisely what matters most.
• 18:22
  SPEAKER_00: Three things to carry forward.
• 18:24
  SPEAKER_00: Ask the second question.
• 18:26
  SPEAKER_00: Treat fluency as a warning sign, not a proxy for depth, and protect the environments where tacit knowledge passes from one human to another.
• 18:35
  SPEAKER_00: Here's my closing thought.
• 18:37
  SPEAKER_00: For centuries, we've had a fantasy that if we could write everything down, we would solve the problem of expertise.
• 18:45
  SPEAKER_00: AI is the ultimate expression of that fantasy, but Polanyi warned us we can know more than we can tell.
• 18:52
  SPEAKER_00: The body knows things the page cannot hold.
• 18:55
  SPEAKER_00: Use AI to augment, accelerate, and assist, but never to substitute for the kinds of knowing that can only be earned through presence, practice, and years inside a domain.
• 19:07
  SPEAKER_00: That's what the manicured knowledge paradox asks us to understand.
• 19:12
  SPEAKER_00: If Mindcast makes you think, subscribe wherever you're listening and leave us a review.
• 19:17
  SPEAKER_00: 60 seconds.
• 19:18
  SPEAKER_00: It helps other curious people find conversations like this one.
• 19:22
  SPEAKER_00: I'm Will.
• 19:23
  SPEAKER_00: This has been Mindcast.
• 19:25
  SPEAKER_00: Stay curious, stay humble, and I'll see you in the next episode.