Latest / AI Fire Daily / #657 Neil: I Audited My Claude Code With Context Engineering Tips

Transcript

#657 Neil: I Audited My Claude Code With Context Engineering Tips · AI Fire Daily

• 0:00
  Human beings have this, uh, fundamental hoarding
• 0:02
  problem. We hoard physical things in our dusty
• 0:06
  garages. Right, or we hold onto old clothes just
• 0:09
  in case. Exactly. And we hoard digital instructions
• 0:12
  in our code, too. We just naturally assume more
• 0:15
  information is always better. But Anthropic just
• 0:19
  proved us completely wrong. They really flipped
• 0:22
  the script on us recently. Welcome to the deep
• 0:24
  dive, everyone. Yeah, we are looking at a great
• 0:27
  stack of notes today. The core topic is known
• 0:29
  as context engineering. We are exploring why
• 0:32
  giving an AI less information works. We will
• 0:35
  look at why it actually makes it smarter. We
• 0:38
  have the hard data proving massive prompts hurt
• 0:40
  performance, and we will walk through a step
• 0:43
  -by -step cleanup guide together. You could use
• 0:45
  it to fix your own coding setup. It is a critical
• 0:48
  topic for anyone using these tools. You really
• 0:50
  need to understand how context works. I have
• 0:54
  to admit something to you, though. I still wrestle
• 0:56
  with prompt drift myself. Wait, really? You still
• 0:59
  find yourself doing that? I absolutely do. You
• 1:02
  get this overwhelming urge to keep adding instructions.
• 1:04
  You just want to cover every single edge case
• 1:07
  imaginable. Yeah. You think the AI gets smarter
• 1:09
  with more details. It is a very natural human
• 1:12
  instinct, honestly. We over explain things when
• 1:15
  we are feeling anxious. We do. But our intuition
• 1:18
  about AI instructions is backwards. Let us look
• 1:21
  at the catalyst for this whole discussion. Anthropic
• 1:23
  fundamentally changed how they handle their system
• 1:25
  prompt. They made a massive adjustment to their
• 1:28
  core architecture recently. And a system prompt
• 1:31
  it is, well, let's define it. It's the core hidden
• 1:33
  instructions guiding how an AI model behaves.
• 1:37
  Right. Anthropic removed over 80 % of those core
• 1:40
  rules. They essentially wiped out the model's
• 1:42
  main rule book. I hear that, but it still feels
• 1:44
  kind of wrong to me. As a developer, my instinct
• 1:47
  is to... document absolutely everything. If I
• 1:50
  don't tell the AI what to do, I worry. I just
• 1:52
  assume it's going to break my code entirely.
• 1:54
  That fear is exactly what Anthropic wanted to
• 1:57
  test. They ran extensive coding evaluations after
• 1:59
  deleting those core rules. The results showed
• 2:02
  no measurable drop in coding performance. That
• 2:05
  feels incredibly counterintuitive at first glance.
• 2:08
  Why wouldn't a model perform worse with fewer
• 2:10
  explicit instructions? It comes down to how these
• 2:13
  language models have evolved. Older models were
• 2:16
  actually quite fragile in their reasoning. Yeah,
• 2:18
  they needed strict, detailed rules to avoid basic
• 2:21
  mistakes. They basically needed their hand held
• 2:23
  through every single step. They absolutely did.
• 2:26
  Take code comment length for a specific example.
• 2:30
  Anthropic used to set strict limits on comment
• 2:32
  length. Because the old models genuinely needed
• 2:35
  that explicit boundary. Right, otherwise they
• 2:37
  would hallucinate and ramble endlessly. But newer
• 2:40
  models do not need that level of hand holding.
• 2:43
  They can infer the rules simply by looking at
• 2:45
  your project. They analyze the existing coding
• 2:47
  style and match it perfectly. Exactly. They rely
• 2:50
  on pattern recognition instead of rigid rules.
• 2:53
  So keeping hundreds of legacy rules is actually
• 2:55
  counterproductive. You are feeding the model
• 2:57
  solutions to problems it already solved. You
• 3:01
  are just wasting your valuable tokens at that
• 3:03
  point. Yeah, and wasted tokens actively lead
• 3:05
  to worse performance overall. And just to clarify,
• 3:08
  tokens are the tiny fragments of words an AI
• 3:11
  uses to process text. Spot on. We have some startling
• 3:15
  data on this specific issue. A comprehensive
• 3:17
  2025 study looked at massive prompt structures.
• 3:21
  What did they find? They found that longer inputs
• 3:24
  actively degrade the model's output. Depending
• 3:26
  on the task, performance dropped by 13 .9%. That
• 3:30
  is a significant drop just from adding extra
• 3:33
  words. It is. And the degradation gets much worse
• 3:36
  in certain scenarios. In some edge cases, performance
• 3:39
  plummeted by up to 85%. That is essentially a
• 3:45
  total system failure. It is a complete breakdown
• 3:47
  of the model's reasoning capabilities. More context
• 3:50
  means the AI processes lots of irrelevant instructions.
• 3:54
  It is fundamentally an attention problem under
• 3:56
  the hood. The architecture has to weigh every
• 3:59
  word against every other word. Exactly. When
• 4:02
  you add garbage, it dilutes the important signals.
• 4:05
  Think about making a simple peanut butter sandwich.
• 4:08
  You already know the basic mechanics of this
• 4:10
  task. Sure, it is a very straightforward everyday
• 4:12
  activity. Now imagine I hand you a 1 ,000 page
• 4:16
  instructional manual. You have to read the entire
• 4:19
  thing before you begin. Oh man, I would probably
• 4:21
  just give up in order to take out. Right. But
• 4:24
  imagine you are forced to read it while working.
• 4:26
  The manual covers the exact optimal angle to
• 4:29
  hold the knife. Wow. It has 50 pages on the history
• 4:32
  of wheat farming. It mandates the precise room
• 4:36
  temperature for spreading peanut butter. That
• 4:38
  is just overwhelming noise distracting from the
• 4:40
  actual goal. Worse than that, you are actively
• 4:42
  holding a sharp knife. You are trying to slice
• 4:45
  bread while reading page 400. Your working memory
• 4:48
  is completely maxed out by the manual. You are
• 4:50
  going to drop the knife or ruin the bread. The
• 4:52
  AI suffers from that exact same cognitive overload.
• 4:55
  It loses the plot because it is reading wheat
• 4:58
  farming history. So is the goal simply to make
• 5:01
  every prompt as short as possible? Not exactly.
• 5:04
  You just need to remove the clutter and keep
• 5:06
  essential guidelines. So don't delete usefulness,
• 5:09
  just stop forcing it to read irrelevant data.
• 5:12
  That perfectly captures the essence of context
• 5:14
  engineering. We know dumping rules into the prompt
• 5:17
  is bad. Which brings us to a major logistical
• 5:20
  catch -22. If dumping all information upfront
• 5:24
  degrades performance, we have a problem. We do.
• 5:27
  How do we give Claude access to powerful development
• 5:29
  tools? If I don't give it instructions, it cannot
• 5:32
  do its job. Right. You want the AI to use complex
• 5:35
  external tools. But... We cannot load massive
• 5:38
  instructions for every tool up front. That would
• 5:41
  trigger the exact cognitive overload we just
• 5:43
  discussed. We would blow out the context window
• 5:46
  immediately. Exactly. And the context window
• 5:49
  is the total amount of text an AI can remember
• 5:51
  at once. We need a way to bypass this memory
• 5:54
  limitation entirely. That is where a concept
• 5:57
  called progressive disclosure comes in. This
• 5:59
  engineering method relies heavily on what are
• 6:01
  called MCP tools. Let's define those quickly
• 6:04
  for the audience. MCP tools are small software
• 6:07
  bridges that let AI read your local files safely.
• 6:10
  They are incredibly useful for coding and deep
• 6:13
  data analysis. They completely change how models
• 6:15
  interact with your local environment. But the
• 6:18
  old way was loading every tool definition instantly.
• 6:21
  You would give the AI the heavy manual for every
• 6:24
  single tool. Even if the current coding task
• 6:26
  did not require them. Progressive disclosure
• 6:29
  changes this underlying mechanism entirely. Think
• 6:32
  of it like giving the AI a simple table of contents.
• 6:36
  Exactly. Instead of loading the whole book up
• 6:38
  front, you load just the titles. The AI knows
• 6:41
  the tools exist in the system, but it waits to
• 6:43
  read about them until they are actually needed.
• 6:46
  The model is smart enough to request the specific
• 6:48
  chapter later. Anthropic tested this specific
• 6:51
  approach in their engineering labs. And what
• 6:53
  were the results? The results from their initial
• 6:55
  tests were genuinely impressive. This method
• 6:58
  used 85 % fewer tokens in the tested example.
• 7:02
  That is a massive reduction in wasted processing
• 7:05
  power. And it directly translated to significantly
• 7:08
  better evaluation scores. Claude Opus 4 went
• 7:11
  from a 49 % score to 74%. That is a staggering
• 7:16
  leap in baseline capability. It is. Claude Opus
• 7:19
  4 .5 saw very similar performance gains. It jumped
• 7:22
  from 79 .5 % to 88 .1%. Whoa. Imagine the efficiency
• 7:29
  of scaling that across billions of queries. It
• 7:32
  completely changes the economics of running these
• 7:34
  massive models. It makes them faster, cheaper,
• 7:38
  and objectively much smarter. But I want to clarify
• 7:41
  the mechanics of this disclosure. Does keeping
• 7:44
  tools unloaded mean the AI forgets they exist?
• 7:47
  Not at all. It sees a lightweight menu of tools
• 7:50
  and downloads instructions later. Right. It sees
• 7:53
  the tool menu, but only downloads the manual
• 7:56
  when asked. It is highly efficient context management
• 7:59
  in action. So we understand the theory and the
• 8:01
  data behind this. Progressive disclosure clearly
• 8:03
  works, and less is absolutely more. Yeah. Now
• 8:07
  we need to translate that into practical application.
• 8:10
  We need to look at your actual daily workflow.
• 8:13
  We really need to apply this to your actual project
• 8:16
  files. Let us look at the standard Claude .md
• 8:20
  file. Many developers use this file to set baseline
• 8:23
  project instructions. The source material is
• 8:26
  very clear on how to handle this. You must keep
• 8:28
  this specific configuration file incredibly lightweight.
• 8:32
  It should only include what Claude cannot easily
• 8:34
  find itself. Right. There are four specific categories
• 8:37
  of information you should keep. First, goals.
• 8:41
  What do you actually want the project to achieve?
• 8:44
  Second, preferences. How do you want Claude code
• 8:47
  to specifically work? Like, preferring functional
• 8:50
  programming over object -oriented programming
• 8:52
  approaches. Exactly. Third, project constraints.
• 8:56
  What are the strict rules and immovable requirements?
• 8:59
  Finally, project -specific knowledge. Details
• 9:02
  Claude cannot figure out just by scanning your
• 9:04
  code base. Wait, I need to push back on this
• 9:07
  a little bit. Sure. Isn't it risky to trust the
• 9:09
  model with unstated rules? If I don't tell it
• 9:12
  my exact coding style up front, what is it looking
• 9:14
  at? It looks at the actual environment you are
• 9:17
  working in. The model scans your folder structures
• 9:19
  and variable naming conventions. Okay, so it
• 9:23
  builds its own context from the environment.
• 9:25
  Yeah. It reads your previous commits to understand
• 9:28
  your historical styling. You just have to structure
• 9:31
  your remaining instructions correctly. You have
• 9:33
  to focus on three core communication pillars,
• 9:36
  result, reason, and constraints. Result is simply
• 9:39
  what you want Claude to finally deliver. Reason
• 9:42
  is why that specific engineering task actually
• 9:44
  matters. And constraints dictate what Claude
• 9:47
  must follow or entirely avoid. Let us use a concrete
• 9:50
  development example to illustrate this clearly.
• 9:53
  Let's say you want to improve your website's
• 9:55
  loading speed. The old way was writing a massive
• 9:58
  complicated prompt. You would list 50 different
• 10:00
  performance optimization techniques manually.
• 10:03
  You would tell it exactly which image files to
• 10:05
  heavily compress. You would write out the exact
• 10:08
  caching logic you wanted implemented. Which just
• 10:11
  wastes tokens and limits the AI's actual problem
• 10:14
  solving. forces the AI to execute instead of
• 10:17
  genuinely thinking. The context engineering way
• 10:20
  is much simpler and faster. Tell Claude the result.
• 10:24
  Dramatically improve the website loading speed.
• 10:26
  Tell it the reason. Users are dropping off on
• 10:29
  mobile devices. And add your constraint. Keep
• 10:31
  the visual design completely unchanged. Then
• 10:34
  you step back and let the machine work. You let
• 10:36
  Claude inspect the complex code base on its own.
• 10:39
  You let it suggest a strategic approach before
• 10:41
  you approve any changes. You are guiding the
• 10:44
  advanced model, not micromanaging its keystrokes.
• 10:47
  Rather than front -loading instructions, you
• 10:49
  are front -loading your actual intent. This philosophy
• 10:52
  also applies to how you handle specialized tasks.
• 10:55
  Move highly specific workflow tasks into dedicated,
• 10:59
  isolated skills. You load those specific skills
• 11:02
  only when they are actually needed. For example.
• 11:05
  You might have complex video editing formatting
• 11:07
  instructions. You should keep those securely
• 11:10
  inside a dedicated video skill. You definitely
• 11:14
  should not load them while building a simple
• 11:15
  website. It sounds obvious, but people leave
• 11:18
  these instructions active everywhere. They bloat
• 11:21
  their workspace with irrelevant data constantly.
• 11:24
  It goes back to that deep fear of prompt drift.
• 11:27
  We just leave things turned on in the background
• 11:28
  just in case. But keeping the file lightweight
• 11:31
  requires some ongoing behavioral maintenance.
• 11:34
  How often should you update this lightweight
• 11:36
  file? You should review it regularly, treating
• 11:39
  it as a dynamic asset that evolves. Treat it
• 11:42
  like a living document. Prune it whenever project
• 11:44
  goals shift. That is the absolute best way to
• 11:47
  maintain optimal context management. We will
• 11:50
  be right back after a quick word from our sponsors.
• 11:52
  And we are back to the deep dive. We have designed
• 11:56
  the ideal lightweight setup in theory. We know
• 11:59
  how to front load intent and use progressive
• 12:01
  disclosure. But to get there, We need to do some
• 12:04
  deep cleaning. We need to diagnose and clear
• 12:07
  out the hidden junk. That hidden junk is actively
• 12:10
  slowing down your system today. It is costing
• 12:12
  you actual money and computational speed. The
• 12:15
  source nodes walk through a brilliant real -world
• 12:18
  audit process. This is incredibly practical for
• 12:20
  anyone currently using cloud code. It all starts
• 12:23
  with one simple, powerful terminal command. You
• 12:26
  just open your command line terminal and run
• 12:28
  slash doctor. That command triggers an automatic
• 12:31
  deep inspection of your setup. It generates a
• 12:34
  comprehensive report showing configuration issues
• 12:37
  and recommendations. The findings from the source's
• 12:40
  real -world audit were honestly fascinating.
• 12:43
  The user's installation was generally healthy,
• 12:45
  but it had hidden baggage. First, the system
• 12:48
  found a completely unused plugin called token
• 12:51
  -weather. It had absolutely zero recorded usage
• 12:54
  in the system logs. It was just sitting there,
• 12:56
  waiting to consume tokens for no reason. The
• 12:59
  audit also found 14 distinct cloud -synced plugins.
• 13:02
  Those synced plugins had a massive combined skill
• 13:05
  listing attached. That implies they might be
• 13:08
  using context silently in the background. The
• 13:10
  audit also caught a bad terminal path configuration
• 13:13
  error. And it found a leftover entirely outdated
• 13:17
  installation executable. That old executable
• 13:19
  was taking up 239 megabytes. It is amazing what
• 13:23
  piles up in the dark when you aren't looking.
• 13:26
  It really is a pervasive issue for most developers.
• 13:28
  It is like checking the trunk of your car. You
• 13:31
  look in there before a long cross -country road
• 13:34
  trip, you find heavy junk you don't need dragging
• 13:36
  you down. Yeah, you have a heavy bowling ball
• 13:39
  and some old textbooks hiding in there. Exactly.
• 13:41
  It ruins your vehicle's overall fuel efficiency.
• 13:44
  That is what these hidden plugins do to your
• 13:46
  context window. Every wasted token costs you
• 13:48
  money and slows down response time. To see the
• 13:51
  exact fuel efficiency, you use the next important
• 13:54
  command. You type slash context directly into
• 13:57
  your terminal prompt. This reveals the precise
• 13:59
  token breakdown of your current active session.
• 14:03
  The numbers from the audit test are very revealing
• 14:05
  to look at. The total context usage was 93 .8k
• 14:10
  tokens. That is about 9 % of the 1 million token
• 14:15
  maximum window. Now, look closely at where those
• 14:18
  tokens were actually going. The core system prompt
• 14:21
  used around 2 .5k tokens. Custom agents used
• 14:26
  about 1 .9k tokens. But the really interesting
• 14:29
  part was the detailed skills breakdown. The developer
• 14:33
  setup had 205 skills actively installed. Those
• 14:38
  active skills were eating up 9 .9k tokens. And
• 14:42
  contrast that heavy load with the available MCP
• 14:45
  tools. They had 614 complex MCP tools available.
• 14:50
  But those hundreds of tools were taking up zero
• 14:52
  tokens. Zero tokens. because they were brilliantly
• 14:55
  unloaded using the progressive disclosure method.
• 14:58
  That is the ultimate mathematical proof of concept
• 15:01
  right there. 600 powerful tools cost you absolutely
• 15:04
  nothing to keep ready. Right. 200 active skills
• 15:07
  cost you nearly 10 ,000 tokens constantly. It
• 15:10
  clearly shows why context engineering is so desperately
• 15:13
  needed today. So the final crucial step is the
• 15:17
  actual cleanup phase. But you cannot just blindly
• 15:20
  delete everything you see in the report. No,
• 15:22
  you need a highly methodical approach to the
• 15:24
  cleanup process. You actually ask Claude to carefully
• 15:27
  audit the setup for you. You give it a very specific
• 15:30
  targeted prompt to investigate further. You ask
• 15:33
  it to identify outdated, duplicated, or totally
• 15:37
  unnecessary instructions. You ask for the exact
• 15:40
  configuration files involved in the cleanup.
• 15:42
  And you importantly ask for a reasonable estimate
• 15:45
  of the token savings. In the test audit, Claude
• 15:48
  found significant potential context savings.
• 15:51
  It estimated saving roughly 6 to 7k tokens per
• 15:54
  session. That is a massive chunk of processing
• 15:57
  power handed back to you. But the most important
• 15:59
  rule is requiring your own manual approval. You
• 16:03
  tell Claude, do not delete anything until I explicitly
• 16:05
  approve. Right, because some instructions look
• 16:08
  unnecessary but actually protect vital workflows.
• 16:11
  You have to manually verify everything before
• 16:13
  committing to the final deletion. And after you
• 16:16
  clean it up, Check slash context one more time.
• 16:19
  You want to verify the estimated savings actually
• 16:21
  materialized in the system. But looking at the
• 16:24
  raw numbers raises a valid strategic question.
• 16:27
  Is the 1 % context usage from those active skills
• 16:30
  really worth deleting? Absolutely. Even a tiny
• 16:33
  amount of irrelevant context pulls focus from
• 16:36
  your main goals. Exactly. Small leaks sink the
• 16:38
  ship. Every wasted token degrades the AI's focus.
• 16:41
  It is all about maintaining that sharp, unyielding
• 16:44
  computational focus. Let's take a step back and
• 16:48
  look at the big picture here. We have covered
• 16:50
  a lot of technical ground in this deep dive.
• 16:52
  We really did. We started with Anthropic bravely
• 16:55
  deleting their core prompt. We explored progressive
• 16:58
  disclosure and managing those lightweight .md
• 17:01
  files. And we walk through the audit process
• 17:04
  using the terminal commands. The core theme here
• 17:06
  is incredibly consistent throughout every single
• 17:08
  topic. Context engineering isn't about simply
• 17:11
  deleting useful necessary information. It is
• 17:14
  about ruthless, uncompromising focus on what
• 17:17
  actually matters. It is the realization that
• 17:19
  actively protecting your AI is crucial. Protecting
• 17:23
  it from reading irrelevant information is absolutely
• 17:26
  paramount. It is just as important as giving
• 17:28
  it the right data up front. You want to give
• 17:31
  the model a clear, unobstructed runway to operate.
• 17:35
  If you use Claude Code, you should take action
• 17:37
  on this today. Open your terminal when you have
• 17:40
  a free moment this afternoon. Run slash doctor.
• 17:43
  and then run slash context immediately after
• 17:46
  that. You might be genuinely shocked at what
• 17:49
  is dragging your AI down. See what hidden legacy
• 17:52
  instructions are burning up your valuable processing
• 17:54
  tokens. You know, reviewing all this data leaves
• 17:57
  me with one final thought. Anthropic proved something
• 18:00
  genuinely profound with their recent system update.
• 18:04
  Removing 80 % of an AI's core rules didn't hurt
• 18:07
  it at all. No, it performed just as well, if
• 18:10
  not better, without that heavy baggage. It raises
• 18:12
  a fascinating, somewhat uncomfortable question
• 18:15
  about ourselves, doesn't it? How many outdated
• 18:17
  system prompts are we running in our own minds?
• 18:20
  How many assumed rules could we delete today
• 18:23
  without losing a single step?