Latest / AI Fire Daily / #656 Neil: How I Built A Karpathy Clone With Claude Code In 5 Steps

Transcript

#656 Neil: How I Built A Karpathy Clone With Claude Code In 5 Steps · AI Fire Daily

• 0:00
  What if you could actually bottle a genius mind?
• 0:02
  Imagine keeping them right there on your desktop
• 0:05
  as a mentor. That is kind of the ultimate dream,
• 0:07
  right? Welcome to the Deep Dive. Today, you and
• 0:10
  I are unpacking a truly fascinating AI experiment.
• 0:13
  Yeah, it's a really good one. A developer built
• 0:15
  a highly customized Claude code agent entirely
• 0:17
  from scratch. They wanted to perfectly clone
• 0:20
  the teaching style of Andre Karpathy. It is a
• 0:23
  profoundly interesting concept to explore. Karpathy
• 0:26
  is obviously a towering figure in our industry.
• 0:28
  I mean, he was one of the founding members of
• 0:30
  OpenAI. He previously led the autopilot engineering
• 0:33
  team over at Tesla. And now he is building his
• 0:37
  own company called Eureka Labs. He is incredibly
• 0:39
  famous for his educational tutorials online.
• 0:42
  Oh, absolutely. His Zero to Hero neural network
• 0:45
  series is basically legendary now. Yeah, he literally
• 0:49
  teaches you how to build AI from absolute scratch.
• 0:52
  Right. He never just throws complicated ideas
• 0:53
  at you blindly. He actually writes the code right
• 0:56
  there on the screen. Exactly. He explains every
• 0:59
  single mechanical step of the process very clearly.
• 1:02
  He even shows you exactly what happens when things
• 1:05
  inevitably break. But, you know, you obviously
• 1:08
  can't just message him when your code breaks.
• 1:10
  No, you definitely can't expect a personal lesson
• 1:12
  from a busy founder. Yeah, he's a little too
• 1:14
  busy for that. So this developer had a slightly
• 1:17
  crazy, highly ambitious idea. They decided to
• 1:21
  structurally build their own André Carpathy clone.
• 1:25
  They wanted a digital mentor that actually checks
• 1:27
  its own work. Right. Instead of an AI confidently
• 1:31
  handing you broken, hallucinated Python code.
• 1:33
  Which happens way too often. We are going to
• 1:36
  explore this entire architectural process today.
• 1:39
  We will look at how they systematically gathered
• 1:40
  the right data. And we'll structure an AI brain
• 1:43
  from the ground up together. We will encode specific
• 1:46
  behavioral rules for the autonomous agent. We
• 1:49
  will force the AI to rigorously verify its own
• 1:52
  work. And finally, we will test the finished
• 1:54
  clone in reality. It is a remarkable technical
• 1:58
  journey. Let's start with gathering the foundational
• 2:01
  brain itself. You clearly can't just tell an
• 2:03
  AI to act like Harpathi. No, you can't just feed
• 2:06
  it a few random technical articles. If you do
• 2:09
  that, you just get a superficial chatbot. Right.
• 2:11
  It will just hallucinate the specific technical
• 2:13
  details immediately. To build a true clone, you
• 2:17
  need his actual literal words. The developer
• 2:20
  systematically collected over 700 ,000 words
• 2:24
  initially. And that massive data set eventually
• 2:27
  grew to over 1 million words. Wow. They pulled
• 2:31
  from four primary public sources across the internet.
• 2:34
  They grabbed his YouTube lectures and his detailed
• 2:37
  educational tutorials. They also pulled his Twitter
• 2:40
  post dating all the way back to 2023. They collected
• 2:44
  long -form technical articles from his personal
• 2:46
  blog, and they meticulously downloaded his public
• 2:50
  GitHub software repositories. It is basically
• 2:53
  like packing a massive suitcase. You stuff it
• 2:56
  with every book, lecture, and note he has ever
• 2:59
  written before a long trip. That is a great analogy.
• 3:02
  They deployed some very specific technical tools
• 3:04
  for this collection phase. They used YouTube
• 3:07
  -Transcript -API to pull the precise video captions.
• 3:11
  They used YT -DLP to grab the necessary metadata
• 3:14
  and formatting. And they also leveraged TwitterAPI
• 3:17
  .io to systematically archive his social media
• 3:20
  posts. Exactly. But they didn't manually download
• 3:23
  all of this data themselves. Clodcode actually
• 3:27
  built the source collection pipeline on its own.
• 3:29
  Wait, really? No. It autonomously pulled the
• 3:32
  data and stored it cleanly for use. Yeah. It
• 3:35
  generated a specific project folder named Carpathy
• 3:38
  -Brain. Inside that architecture, it established
• 3:41
  a dedicated raw data directory. So the AI wrote
• 3:45
  a Python script to gather everything entirely
• 3:48
  automatically. It is an incredibly efficient
• 3:50
  way to handle data ingestion. But scraping a
• 3:53
  million words from the internet can be incredibly
• 3:55
  messy. I mean, you might pull the same exact
• 3:58
  YouTube transcript three different times. That
• 4:01
  is exactly why they created a CriticalManifest
• 4:03
  .json file. Ah, okay. This specific manifest
• 4:07
  file acts as a strict inventory control system.
• 4:10
  It rigorously tracks the exact word counts for
• 4:13
  every single document. It completely prevents
• 4:16
  any duplicate files from corrupting the dataset.
• 4:18
  Wait, does it just scrape everything blindly
• 4:21
  and guess the missing parts? No. The strict manifest
• 4:24
  prevents the AI from inventing missing data.
• 4:27
  So it acts like a strict bouncer for incoming
• 4:30
  files. Got it. It is a crucial safeguard for
• 4:32
  data integrity. So we have a million words sitting
• 4:38
  on a hard drive now. But raw data is basically
• 4:42
  just a chaotic, messy pile. Yeah, it's pretty
• 4:44
  much useless in that state. To make it genuinely
• 4:47
  useful, the AI needs rigid, underlying structure.
• 4:51
  It needs to know how to logically connect the
• 4:53
  conceptual dots. Exactly. It is like stacking
• 4:57
  Lego blocks of data into a perfectly organized
• 5:00
  library. You cannot just dump raw text files
• 5:03
  into an AI prompt. The developer converted everything
• 5:06
  into an interconnected markdown wiki. This architecture
• 5:10
  was actually directly inspired by Karpathy's
• 5:12
  own wiki framework. It was. The wiki meticulously
• 5:16
  grouped the raw data by its underlying meaning.
• 5:18
  So it deliberately linked related technical concepts
• 5:21
  together on specific internal pages. Right, and
• 5:24
  it cleanly grouped various debugging approaches
• 5:26
  and broad engineering principles together. They
• 5:30
  also established a very important project level
• 5:32
  memory file. Yeah, a configuration file specifically
• 5:35
  called Claude .md. This file effectively establishes
• 5:38
  the fundamental rules of engagement here. It
• 5:41
  essentially tells the AI exactly how to navigate
• 5:43
  the library. The rules explicitly force the AI
• 5:46
  to read quick summaries first. These brief summaries
• 5:50
  are stored in a file called hot .md. It also
• 5:53
  rigorously forces the AI to trace factual claims
• 5:56
  back. It absolutely must link its statements
• 5:58
  back to the raw sources. Right, and it strictly
• 6:01
  separates documented facts from its own interpretations.
• 6:04
  That is huge. The agent cannot pretend an inferred
• 6:08
  opinion is a documented historical fact. It has
• 6:10
  to clearly state whenever it is making an educated
• 6:13
  guess. It even maintains a special directory
• 6:16
  just for uncertain findings. If concrete evidence
• 6:19
  is missing, it marks the claim as unverified.
• 6:23
  It is a deeply scientific approach to personal
• 6:26
  knowledge management. It fundamentally prevents
• 6:28
  the AI from making sweeping, unverified technical
• 6:31
  assumptions. Wait, if it is just reading Markdown,
• 6:34
  how does it handle contradicting sources over
• 6:37
  time? It preserves the disagreements and highlights
• 6:40
  the original publication dates. So it treats
• 6:43
  changing opinions like a historical timeline.
• 6:46
  Makes sense. Yeah. Human experts constantly evolve
• 6:49
  their technical opinions over time. So with the
• 6:52
  comprehensive library built, we need a competent
• 6:54
  librarian. We need an autonomous AI that actually
• 6:58
  thinks like the master. We need to explicitly
• 7:00
  give it his personality and specific skills.
• 7:03
  The developer cleverly used Claude code for this
• 7:06
  critical next step. They specifically instructed
• 7:08
  it to extract seven distinct thinking rules.
• 7:11
  It pulled these behavioral rules directly from
• 7:14
  the compiled Markdown Wiki. These extracted rules
• 7:16
  include several classic Karpathy educational
• 7:19
  stables. Like, you always build small examples
• 7:22
  to truly understand complex concepts. You absolutely
• 7:25
  must predict results before running any new code.
• 7:28
  You also deliberately show failures to explain
• 7:30
  why they actually happen. You must verify all
• 7:33
  technical claims using evidence and real tests.
• 7:37
  And you should always prefer simple solutions
• 7:39
  over unnecessary structural complexity. We just
• 7:42
  easier said than done, obviously. I mean, I still
• 7:44
  wrestle with prompt drift myself, honestly. Oh,
• 7:47
  we all do. It's a huge problem. So locking in
• 7:50
  these specific behavioral rules at the system
• 7:52
  level is brilliant. It physically stops the AI
• 7:55
  from wandering off into generic answers. It keeps
• 7:57
  the autonomous aging incredibly focused and highly
• 8:00
  disciplined, structurally. This extraction process
• 8:04
  resulted in a dedicated Claude code sub agent.
• 8:08
  A sub agent is basically a highly specialized
• 8:11
  tightly -focused AI assistant. Exactly. It also
• 8:15
  created a reusable internal skill called slash
• 8:18
  carpathy dash teach. That specific command lets
• 8:22
  the user summon the personalized clone anytime.
• 8:25
  But the sub -agent is strictly instructed regarding
• 8:27
  its own internal identity. It absolutely must
• 8:30
  identify itself as being merely carpathy -inspired.
• 8:33
  It can never falsely claim to be the actual human
• 8:36
  being. It is a functional clone, but it inherently
• 8:39
  knows it is a clone. It is explicitly programmed
• 8:42
  to be entirely transparent about its artificial
• 8:44
  nature. Can this agent access unrelated projects
• 8:48
  on my local machine? No. All project operations
• 8:51
  remain entirely local and strictly contained.
• 8:54
  Good. So it won't rummage through my personal
• 8:57
  folders. Reassuring. Security and isolation are
• 8:59
  built directly into the agent workflow. They
• 9:03
  might write an entire script without checking
• 9:05
  if it works. Yeah. They might write an entire
• 9:07
  script without checking if it works. How do we
• 9:11
  physically force it to be academically rigorous?
• 9:14
  Well, we have to make it physically prove its
• 9:16
  work every time. This specific mechanism acts
• 9:19
  as a firm blocker for the AI. If the agent writes
• 9:22
  or modifies any actual code files, it cannot
• 9:25
  complete the designated task until it literally
• 9:28
  runs it. If it stubbornly tries to bypass this
• 9:30
  mandatory testing step, the software hook actively
• 9:33
  blocks it with a very specific message. The system
• 9:36
  message strictly says, Run it before you say
• 9:40
  it works. It is an incredibly powerful concept
• 9:43
  for personal software development. Whoa. Imagine
• 9:46
  having an automated mentor that forces you to
• 9:49
  be rigorous at 2am. It changes everything. Human
• 9:52
  developers naturally get fatigued and start skipping
• 9:55
  vital testing steps. This structural hook forces
• 9:58
  a very specific, unbreakable workflow on the
• 10:01
  AI. The mandatory sequence is predict, build,
• 10:05
  run, compare, and then explain. So it fundamentally
• 10:08
  must guess the output before it executes the
• 10:11
  script. Right. It cannot just run the code and
• 10:14
  act surprised by errors. It has to explicitly
• 10:16
  state its underlying hypothesis beforehand. The
• 10:19
  developer carefully ran 16 simulation tests on
• 10:22
  this specific hook. They desperately wanted to
• 10:24
  ensure it wouldn't create infinite automated
• 10:27
  loops. Yeah. You definitely don't want the AI
• 10:30
  endlessly retrying deeply broken code, burning
• 10:32
  through API tokens while stuck in an inescapable
• 10:35
  logic loop. The simulation's happily proved it
• 10:37
  handles failed tests safely. It inherently knows
• 10:40
  when to stop trying and simply ask for help.
• 10:43
  It clearly reports the unresolved technical problems
• 10:46
  back to the human user. Which is great. What
• 10:48
  happens if I just ask a conceptual question without
• 10:51
  writing code? The hook simply ignores it and
• 10:53
  allows the agent to answer normally. The blocker
• 10:56
  only triggers when actual code is modified, elegantly
• 10:59
  designed. It doesn't get in the way of a normal
• 11:01
  conversation unnecessarily. The strict rules
• 11:05
  are set and the data is heavily verified. The
• 11:07
  execution stop hook is actively running in the
• 11:10
  background. It is finally time for the ultimate
• 11:12
  comprehensive final test. The developer directly
• 11:16
  prompted the slash carpathy dash teach skill
• 11:19
  in their terminal. They asked it to independently
• 11:22
  build a small BPE tokenizer. They specifically
• 11:25
  wanted it built entirely from scratch in Python.
• 11:28
  And let's quickly clarify that term for a moment.
• 11:30
  A tokenizer chops text into small pieces an AI
• 11:33
  can read. Right. It is a completely fundamental
• 11:36
  part of modern large language models. Byte Pair
• 11:39
  Encoding is how they actually process our human
• 11:42
  language. The autonomous agent followed Carpathy's
• 11:45
  exact documented methods perfectly. It initially
• 11:48
  explained the basic underlying idea in highly
• 11:50
  beginner -friendly language. Then it clearly
• 11:53
  defined what a successful tokenizer should actually
• 11:56
  functionally do. It strategically built the absolute
• 11:59
  smallest working implementation first. It clearly
• 12:02
  predicted the exact text output before running
• 12:04
  the new code. Then it actively ran the code and
• 12:07
  compared the actual real results. It flawlessly
• 12:10
  passed 8 out of 8 required technical tests successfully.
• 12:14
  It successfully handled basic English text and
• 12:17
  highly complex Unicode characters. It even correctly
• 12:20
  processed visual emojis without throwing any
• 12:23
  unexpected errors. It also deliberately showed
• 12:26
  a highly meaningful failure case during testing.
• 12:29
  It explicitly explained what went wrong and exactly
• 12:31
  how to fix it. It properly cited relevant educational
• 12:34
  sources from the compiled Karpathy knowledge
• 12:36
  base. Right, and the stop hook actively worked
• 12:39
  in a real interactive session. The agent lazily
• 12:42
  tried to finish without testing its code exactly
• 12:44
  once. The background hook instantly blocked it
• 12:47
  and forced it to run tests. This completely validates
• 12:50
  the entire crazy experiment for me, conceptually.
• 12:52
  The digital clone actually functions exactly
• 12:54
  as they originally intended. It strictly follows
• 12:57
  the rules and actively catches its own programming
• 12:59
  mistakes. It conclusively proves the underlying
• 13:01
  software architecture is solid and highly reliable.
• 13:05
  Can you easily update the clone when Carpathi
• 13:08
  releases a new video? Yes, you just feed the
• 13:10
  new content into the existing pipeline. It digests
• 13:12
  the new material and keeps on growing. Fascinating.
• 13:15
  It effectively acts as a continuously evolving
• 13:17
  digital brain. We will explore what this means
• 13:21
  for the future shortly, sponsor. We have covered
• 13:25
  a tremendous amount of conceptual ground today.
• 13:27
  Let's take a moment to step back and look at
• 13:29
  the big picture. We really need to carefully
• 13:31
  distill the core insight here. This deeply fascinating
• 13:35
  experiment isn't just about cloning Andrej Karpathy
• 13:38
  specifically. It is fundamentally about a replicable,
• 13:42
  scalable framework for artificial AI mentorship.
• 13:46
  You meticulously combine a highly curated knowledge
• 13:49
  base like the Markdown Wiki. You structurally
• 13:52
  add strict behavioral principles via the dedicated
• 13:55
  Claude sub -agent. You rigorously enforce academic
• 13:59
  integrity with a mandatory execution stop hook.
• 14:02
  Literally anyone can build an autonomous AI clone
• 14:04
  of their favorite expert. You simply need their
• 14:07
  public data and their deeply documented methods.
• 14:11
  It can fundamentally act as a highly personalized,
• 14:14
  endlessly patient mentor. Seriously, think about
• 14:17
  whose specific brain you would want to download.
• 14:20
  Who would you actively turn into a custom, personalized
• 14:24
  clot agent? It could dramatically change how
• 14:27
  we learn complex technical skills forever. You
• 14:30
  are simply no longer limited by your geographic
• 14:34
  access to experts. If we can mathematically clone
• 14:37
  the methodologies of our professional heroes,
• 14:39
  what actually happens when we use these same
• 14:42
  software tools on ourselves? That is wild to
• 14:45
  think about. Could you eventually pair program
• 14:47
  with an idealized, perfectly rigorous version
• 14:50
  of yourself? Imagine seamlessly pair programming
• 14:53
  with a highly focused version of yourself. A
• 14:56
  perfectly rigorous version that literally never
• 14:58
  gets tired or lazy. That is definitely something
• 15:03
  to deeply mull over tonight. Thank you for joining
• 15:06
  us on this deep dive. Keep questioning everything
• 15:09
  around you.