Latest / AI Fire Daily / #513 Neil: AI Loop Keeps Working On Your Codebase While You Sleep
Transcript
- 0:00You're staring blankly at your computer screen.
- 0:02It's early in the morning. You just type the
- 0:04word continue again. This is the fifth time you
- 0:07typed it today. The AI stopped at the exact same
- 0:10spot. It sits there waiting for your explicit
- 0:13permission. Beat. It completely shatters your
- 0:16deep state of focus. It really is an incredibly
- 0:18frustrating loop. We are constantly moving away
- 0:22from manual prompting, but that habit is hard
- 0:24to break. Right. So let's unpack this together
- 0:27right now. Welcome to our deep dive for today.
- 0:30We're exploring a fascinating new workflow concept.
- 0:33It's called loop engineering for autonomous AI
- 0:36work. This represents a massive shift in human
- 0:38computer interaction. It completely changes how
- 0:40we build modern software. It really is a game
- 0:43changer for developers. Today, we'll unpack the
- 0:45underlying mechanics of AI loops. We'll explore
- 0:48five real -world agentic loops you can copy.
- 0:51We'll also discuss the harsh realities involved
- 0:53here. Specifically, we'll look at spiraling API
- 0:56token costs. API token costs are the digital
- 0:59currency you pay for AI processing. It's a profoundly
- 1:02important topic for modern developers. It basically
- 1:05changes how we think about autonomous digital
- 1:08labor. You're essentially... managing a tiny
- 1:11digital worker. I think a lot about how we interact
- 1:14with machines. Beat, we basically treat standard
- 1:17AI like a timid intern. This intern might be
- 1:20incredibly brilliant at complex tasks, but they
- 1:23need your approval for every single keystroke.
- 1:26Yeah, that's a perfect way to frame the relationship.
- 1:28We're constantly babysitting the artificial intelligence.
- 1:30all day long. We have to hold its hand through
- 1:33every single step. But an AI loop fundamentally
- 1:36changes that entire dynamic. It's much more like
- 1:38a robotic Roomba. You just let the Roomba map
- 1:40the entire floor first. It stops automatically
- 1:42when the room is fully clean. What's fascinating
- 1:45here is the core underlying issue. The primary
- 1:48problem isn't the AI's actual coding capability.
- 1:51The AI handles the complex coding work incredibly
- 1:54easily. The real problem is it just doesn't know
- 1:56when to stop. I still wrestle with prompt drift
- 1:58myself. You try giving the AI too many sequential
- 2:01instructions, it eventually just loses the main
- 2:03plot completely. The context window gets too
- 2:06crowded with complex logic. And a context window
- 2:09is the short -term memory an AI uses. Right.
- 2:12It just forgets what it was originally doing.
- 2:14That's exactly why these AI loops are so incredibly
- 2:17powerful. An AI loop simplifies the entire process
- 2:22significantly. It basically requires two fundamental
- 2:25things to run properly. First, you need a very
- 2:27specific, clearly defined goal. The system absolutely
- 2:31needs to know when the job is actually done.
- 2:33Right. And the second is explicit permission.
- 2:36It needs permission to check. fix, and retry
- 2:39automatically. You don't type anything between
- 2:41the steps at all. The loop handles the entire
- 2:43iterative cycle on its own. It isn't just some
- 2:46magic one -click button, though. It's a specific
- 2:49prompt paired with a strict stop condition. You
- 2:52have to design those boundaries very carefully
- 2:54beforehand. Yeah, this stop condition is the
- 2:56most critical piece. There are three main triggers
- 2:58that usually start these loops. Manual triggers
- 3:01are simply started directly by you in the terminal.
- 3:03Scheduled triggers run automatically at specific
- 3:06preset time. they might run every single night
- 3:09at midnight while you sleep. Exactly. And action
- 3:12-based triggers run after a specific software
- 3:15event occurs. Like someone opening a pull request
- 3:18in your repository system. A pull request is
- 3:21a formal proposal to merge new code changes.
- 3:24The goals themselves generally fall into two
- 3:26distinct categories. Verifiable goals are measured
- 3:29by hard mathematical numbers. LLM as judge goals
- 3:33mean the AI subjectively decides if it's good
- 3:35enough. We'll explore both of those distinct
- 3:38categories today. Matthew Berman actually created
- 3:41something called the Loop Library. He runs an
- 3:43AI loop on his production errors daily. It autonomously
- 3:47reads the server logs for anything actionable.
- 3:49It fixes the error and rigorously verifies the
- 3:52code fix. Then it opens a pull request right
- 3:54there in the system. And if it finds absolutely
- 3:56nothing wrong, it just stops? It doesn't invent
- 3:58random tasks just to look busy? If the AI runs
- 4:01completely on its own, how do we trust it doesn't
- 4:04just invent fake work to look busy? Well, clear
- 4:07boundaries and strict stop conditions prevent
- 4:09that exact scenario. If it sees no actionable
- 4:13errors, the prompt tells it to halt immediately.
- 4:16There is no ambiguity. So with strict boundaries,
- 4:18it simply stops when there's nothing to fix.
- 4:20Yes. It doesn't hallucinate fake problems just
- 4:23to seem productive. It simply stops working and
- 4:26waits patiently for tomorrow. Let's move into
- 4:28the verifiable loops right now. These rely heavily
- 4:31on pure speed and system stability. They use
- 4:34hard numbers and explicit machine -readable server
- 4:37logs. We're talking about highly objective, measurable
- 4:40success metrics here. Verifiable goals are always
- 4:44much easier to trust entirely. The fuzzier the
- 4:47goal, the harder it is to trust the AI. Hard
- 4:50numbers leave absolutely no room for subjective
- 4:53interpretation. The first one is the sub -50
- 4:55millisecond page load loop. The ultimate goal
- 4:57here is one single crystal clear number. Every
- 5:00single web page must load in under 50 milliseconds.
- 5:04That's blazing fast for any modern web application
- 5:07today. The AI continuously measures how long
- 5:09each page takes. It hunts for whatever is slowing
- 5:12the specific code down. It looks for heavy unoptimized
- 5:14images or slow database queries. It also looks
- 5:17for extra JavaScript that delays the page loading.
- 5:19Yeah, it identifies the bottleneck and writes
- 5:21a targeted fix autonomously. Then it immediately
- 5:25remeasures the speed to verify the improvement.
- 5:27It keeps repeating this cycle until every page
- 5:30hits the strict target. The author of our source
- 5:33ran this exact loop himself. He used a powerful
- 5:35tool called Claude Code for this experiment.
- 5:38Claude came up with three specific optimization
- 5:41steps entirely on its own. First, it created
- 5:44a strict baseline for the original code file.
- 5:46The file was over 112 ,000 bytes in total size.
- 5:50The initial load time was sitting at roughly
- 5:520 .7 milliseconds. Then the AI took its very
- 5:55first optimization action. It actually dropped
- 5:57the Google Fonts content delivery network entirely.
- 6:01It switched the code to use local system fonts
- 6:03instead. Which is incredibly smart. Local fonts
- 6:05eliminate the need for extra network requests.
- 6:08Then it went ahead and minified the CSS code
- 6:10files. Minified CSS means removing empty spaces
- 6:13to make code files smaller. It squeezed the file
- 6:16down to its absolute smallest possible footprint.
- 6:18Right. The final file size dropped to roughly
- 6:21108 ,000 bytes overall. The load time successfully
- 6:25hit 0 .6 milliseconds on the server. Every single
- 6:29page comfortably came in under the 50 millisecond
- 6:32mark. It actually achieved the explicit goal
- 6:35perfectly without human intervention. Matthew
- 6:38Berman ran his version of this loop for nearly
- 6:4150 minutes. It just... quietly chipped away at
- 6:44the code -based performance constantly. This
- 6:46specific loop is best for existing stable web
- 6:49pages. It's definitely not meant for sites currently
- 6:52under heavy construction. If the site is changing
- 6:54constantly, the performance benchmark keeps moving.
- 6:57The AI will end up chasing a constantly moving
- 6:59target indefinitely. You have to give it a completely
- 7:01stable foundation to optimize. Exactly. The second
- 7:04verifiable loop is the production error sweep.
- 7:06It reviews your live production logs for any
- 7:08actionable software errors. It traces those exact
- 7:11errors down to their underlying root causes.
- 7:13It writes the necessary fix and thoroughly verifies
- 7:16the code changes. Finally, it opens a pull request
- 7:19for a human developer to review. It repeats this
- 7:21exact cycle until the server logs are completely
- 7:23clean. I have to be brutally honest with you
- 7:25right now. Letting AI automatically fix live
- 7:28production code feels incredibly wild. It's honestly
- 7:32like letting a ghost drive your car to sex islands.
- 7:35you're handing over the steering wheel to an
- 7:38invisible entity. It definitely requires a massive
- 7:41leap of faith at first. But remember those strict
- 7:44algorithmic boundaries we discussed earlier today.
- 7:47It only acts on explicit documented errors found
- 7:50in the logs. It isn't just rewriting your entire
- 7:53system architecture for fun. What happens to
- 7:55the error sweep loop if a company's logging is
- 7:57just too thin or vague? The loop will completely
- 8:00fall apart almost immediately. Thin logging gives
- 8:03the AI absolutely nothing concrete to actually
- 8:06investigate. That's exactly why doing a logging
- 8:09coverage pass beforehand is absolutely crucial.
- 8:12You have to give the AI a clear map to follow.
- 8:14Right. Poor logs mean the AI has absolutely nothing
- 8:16to actually investigate. Exactly. It needs rich,
- 8:19detailed data to find the true root cause. Without
- 8:22highly descriptive logs, it's flying completely
- 8:25blind in the dark. Here's where it gets incredibly
- 8:28interesting for us today. We're leaving the safe
- 8:30comfort of hard, verifiable numerical metrics.
- 8:34We're entering much fuzzier, inherently subjective
- 8:37territory right now. This is the LLM as judge
- 8:40category of autonomous AI loops. The AI evaluates
- 8:44its own work without using strict, hard metrics.
- 8:47It relies entirely on its own semantic understanding
- 8:50of the given task. The third loop is widely known
- 8:52as the dock sweep. You basically give the AI
- 8:56a recurring nightly maintenance job. It must
- 8:58keep documentation in perfect sync with codebase
- 9:01changes. It meticulously reviews the entire codebase
- 9:04every single night. It compares the raw code
- 9:07against the current written documentation. It
- 9:09systematically finds any sections that have gone
- 9:11completely stale. It thoughtfully updates those
- 9:14sections and verifies the text changes itself.
- 9:17Then it opens a pull request for a human review
- 9:19tomorrow. With a standing no -push instruction,
- 9:21it commits the files locally instead. A no -push
- 9:24instruction stops it from publishing code live
- 9:27automatically. It keeps the changes safely isolated
- 9:29on your own local machine. The author ran this
- 9:32specific loop on his own machine recently. Claude
- 9:35actually flagged a completely unrelated PDF document
- 9:39almost immediately. It instantly recognized the
- 9:42PDF wasn't relevant and set it aside. It didn't
- 9:45waste expensive compute time trying to parse
- 9:47a random manual. Then it wrote a brand new re
- 9:50-adme file from scratch. It carefully covered
- 9:52the project layout and the entire server setup.
- 9:55It even documented those 50 millisecond font
- 9:58optimizations we discussed earlier. Then it reran
- 10:01the code base benchmark to confirm nothing broke.
- 10:04It committed the final changes locally, exactly
- 10:06as it was instructed. This specific loop fits
- 10:09teams that ship new code very frequently. Documentation
- 10:12quietly falls behind when you move at extremely
- 10:15high speeds. Developers almost never want to
- 10:17stop and update the text files. Yeah, but if
- 10:19your code base barely changes, This loop basically
- 10:22does nothing. It just checks the files and gracefully
- 10:24goes right back to sleep. The fourth loop is
- 10:26a massive SEO visibility loop. SEO stands for
- 10:30search engine optimization for modern websites.
- 10:32This is how you ensure people can actually find
- 10:34your web content. This loop autonomously audits
- 10:37crawlability, indexation, page titles, and internal
- 10:41links. It basically checks how easily search
- 10:43engines can read your entire site. It closely
- 10:46examines structured data and answer -first content
- 10:48formatting. It's like stacking Lego blocks of
- 10:51data. Beat. The AI methodically snaps unstructured
- 10:54information into a rigid structure. It builds
- 10:57something that search engines can easily read
- 10:59and understand. It systematically ranks all the
- 11:01identified gaps by their expected impact. It
- 11:04fixes the absolute highest leverage issue it
- 11:06can possibly find first. Then it reruns comprehensive
- 11:09audits across major search engines and AI answer
- 11:12engines. It keeps going until no critical technical
- 11:15issues remain unresolved. Beat woe. beat. Imagine
- 11:19scaling to a billion queries. It's a staggering
- 11:22amount of unstructured data to process. This
- 11:24loop actually covers eight separate technical
- 11:27ranking factors continuously. It's an incredibly
- 11:29complex web of interconnected metrics to balance
- 11:32correctly. Because of that complexity, it sits
- 11:34heavily in the token usage category. You should
- 11:37generally run this one weekly, not as a daily
- 11:39task. Search engine crawl results and AI citations
- 11:42shift very slowly anyway. Running it every single
- 11:45day would largely be a waste of resources. Isn't
- 11:47it inherently risky to let the AI be the judge
- 11:50of its own SEO fixes? Well, it is incredibly
- 11:52risky in this specific Highly subjective context.
- 11:56That's exactly why the verify step requires such
- 11:59intense preparation beforehand. You must strictly
- 12:02record exact target queries before it even starts.
- 12:05You need absolute benchmark methods securely
- 12:08locked in place first. You have to lock down
- 12:10the baseline metrics before letting it judge.
- 12:12Yes. Without a rigid, predefined baseline, it
- 12:15might optimize for the wrong things entirely.
- 12:18You absolutely have to measure against that exact
- 12:20same set after every fix. We're pushing LLM as
- 12:25Judge to its absolute functional limit now. We're
- 12:28closely examining the full product evaluation
- 12:30loop. This is the fifth and final loop we are
- 12:32covering today. This is easily the most open
- 12:34-ended loop of them all. It requires the AI to
- 12:37understand complex user intent and product functionality.
- 12:40Before it even touches the live site, it prepares
- 12:42meticulously. It spins up a local replica of
- 12:46your live production environment. It designs
- 12:48highly realistic test scenarios covering every
- 12:51major software feature. It explicitly defines
- 12:54strict pass and fail criteria for each scenario.
- 12:57It runs all those complex scenarios under the
- 12:59exact same conditions. It essentially simulates
- 13:02thousands of users interacting with your application
- 13:05simultaneously. It carefully records the evidence,
- 13:08logs any failures, and fixes root causes. Then
- 13:11it systematically reruns the affected scenarios
- 13:13over and over again. It basically repeats this
- 13:16intense cycle until every scenario holds against
- 13:19the bar. Matthew Berman actually used a variant
- 13:21of this on his own product. He had the AI generate
- 13:24100 different highly complex test questions.
- 13:27These were specific test questions for a feature
- 13:30involving AI responses. Each response got judged
- 13:33heavily for factual accuracy and proper sourcing.
- 13:36Whatever fell short of the strict standard got
- 13:38fixed by the loop immediately. There's absolutely
- 13:40no single metric deciding pass or fail here.
- 13:43The AI has to subjectively judge quality against
- 13:47your preset bar. It requires a very advanced
- 13:50reasoning model to work correctly. Which brings
- 13:53us to the very dangerous traps involved here.
- 13:55You must absolutely not use loops for brand new
- 13:58software features. Do not use them for things
- 14:00built completely from scratch today. Right. A
- 14:02task with nothing yet to measure will predictably
- 14:05fail completely. Building a brand new feature
- 14:07from scratch is the clearest example possible.
- 14:10The AI needs a solid foundation to iterate upon
- 14:12successfully. Building a brand new feature from
- 14:14scratch with an AI loop seems like trying to
- 14:17build a house blindfolded. to sex silence. You
- 14:20have absolutely no idea if the walls are even
- 14:23straight. That's a very accurate and genuinely
- 14:25terrifying visual analogy. A loop can run in
- 14:28the totally wrong direction for hours. Without
- 14:31existing benchmarks, it genuinely never knows
- 14:33it is off track. It will confidently build a
- 14:36completely broken system for you. Loops definitely
- 14:38work best on something that already exists today.
- 14:40They work beautifully on things that just need
- 14:42minor iterative improvements. And that naturally
- 14:45leads us to a very harsh financial reality. The
- 14:48computational costs associated with these loops
- 14:51add up incredibly fast. You're paying for every
- 14:54single piece of text the AI processes. The loop
- 14:57runs through many complex data -heavy steps in
- 14:59a row. The author ran several of these loops
- 15:01himself just recently. He quickly hit his daily
- 15:04limits on Cloud Code Pro and Codex Pro tiers.
- 15:07His API usage climbed drastically faster than
- 15:10he originally expected. The AI consumes context
- 15:13continuously as it iterates through the entire
- 15:15code base. If it's just running quietly in the
- 15:17background, how fast can these API costs actually
- 15:20spiral? Terrifyingly fast. Because it works continuously
- 15:23without pausing, you can hit a daily usage limit
- 15:26in just a few dozen minutes. It processes thousands
- 15:29of tokens every single second it runs. Wow. So
- 15:31you can burn a daily token limit in just minutes.
- 15:34Exactly. A loop that runs long eats through massive
- 15:37token usage quickly. It keeps working the entire
- 15:39time without taking any human breaks. It rereads
- 15:42the entire context history during every single
- 15:44iterative step. You absolutely must watch the
- 15:47cost closely those first few times. OK, let's
- 15:49pull all of this complex information together
- 15:51now. AI loops transform simple, single task chatbots
- 15:55into agentic workflow powerhouses. We're moving
- 15:58from simple text generation to autonomous digital
- 16:01labor. The fundamental secret isn't some hidden,
- 16:05highly complex coding magic. It is simply a clear
- 16:08goal paired with explicit permission to loop.
- 16:10You give the machine clear boundaries and get
- 16:13out of the way. You should always start your
- 16:15experiments with verifiable hard numbers first.
- 16:18Things like simple page load times are a perfectly
- 16:20safe starting place. You can clearly measure
- 16:22if the AI is succeeding or failing instantly.
- 16:25Master those objective metrics before moving
- 16:27to open -ended subjective scenarios. The LLM
- 16:30as judge loops require a much deeper understanding
- 16:32of prompt engineering, and you must always watch
- 16:35those API token costs incredibly closely. This
- 16:38raises a massive, almost existential philosophical
- 16:41question for our future. If an AI can now autonomously
- 16:44identify hidden production errors, right, and
- 16:47if it can write the fix, verify it, and push
- 16:49the code, beat. At what point does a software
- 16:52developer transition from being a coder to simply
- 16:55being a manager of AI agents? And what does that
- 17:00practically mean for junior developers trying
- 17:02to learn the ropes? It fundamentally changes
- 17:04the entire modern career trajectory for software
- 17:06engineers. Developers might become high -level
- 17:09system architects much earlier in their careers.
- 17:12They'll spend less time typing and more time
- 17:14orchestrating complex systems. the true skill
- 17:16shifts from writing syntax to designing flawless
- 17:20verification loops. It really does challenge
- 17:22everything we currently know about coding. I
- 17:24highly encourage you to try implementing just
- 17:27one loop today. Try setting up a simple, highly
- 17:29verifiable loop this week for yourself. See how
- 17:32it radically changes your own daily development
- 17:34workflow. It might just save you a massive amount
- 17:36of tedious manual typing. It frees your mind
- 17:39to focus on the much bigger architectural picture.
- 17:42You'll never have to babysit the timid intern
- 17:44ever again. and just let the robotic Roomba clean
- 17:46the entire floor autonomously. Thank you for
- 17:49joining this deep dive with us today. Audi -Horo
- 17:52music.