Latest / Braid / The Agent That Dreams Between Sessions

Transcript

The Agent That Dreams Between Sessions · Braid

• 0:04
  lenar: Picture an agent that starts a job at nine in the morning and is still going at nine at night, with no human checking in and no chat window open — just a process working through a task somewhere in a datacenter. What has to be true about the software around that model for you to trust it? What happens when the container crashes at hour seven? Who holds the credentials while nobody is looking? Those questions are the spine of today's episode, because Anthropic just gave a surprisingly detailed public answer to all three.
• 0:34
  damra: And the answer is more interesting than the usual reliability talk, because part of it involves the agent going to sleep and dreaming. Literally — there is a process they call offline dreaming, and we should get into what it actually does, because it fixed a bug that I find weirdly moving.
• 0:50
  lenar: We will. So here's the route for today, Thursday, July twenty-third. The lead is a recorded talk from the AI Engineer conference — Lance Martin from Anthropic, posted yesterday afternoon, on how they build the machinery around Claude for long-horizon tasks. Then a reported Guardian interactive on how working software engineers are adapting to all of this, which is the human mirror of the same story. After that, a run of research. There's a memory paper with a result I think a lot of teams need to hear, a big claim about synthetic training data, a payment-integration benchmark from Alipay, and two robotics papers that have independently arrived at problems coding agents already know well.
• 1:33
  damra: A research-heavy day — the pool was thin on product and policy news, so we'll keep the papers at their actual size rather than inflating them. But the Martin talk deserves the full treatment.
• 1:44
  lenar: So the talk is called Claude for Long-Horizon Tasks, and Martin opens with a capability timeline that explains why this architecture exists at all. In 2024, with Claude 3 Opus, a model could hold a coherent task for maybe ten to twenty minutes — chat and autocomplete were the viable products. Through 2025, with Sonnet 4.6 and Opus 4.7, that stretched to around an hour, which is where synchronous coding agents live. And then starting in April of this year, Anthropic's Mythos-class models pushed past twelve hours on a single task.
• 2:21
  damra: And twelve hours changes the engineering problem completely. At one hour, if the session dies, you shrug and restart — you lost an hour. At twelve hours, losing the session is expensive enough that crash survival becomes the first requirement, not a nice-to-have. And the product flips from synchronous to asynchronous, because no human is going to sit and watch a terminal for twelve hours. The economics of the whole product category pivot on that one threshold.
• 2:49
  lenar: Right, and his core architecture follows from that threshold. He splits the system into three parts. The harness is a stateless orchestration process — he calls it the brain. The session is an append-only event log, and that log is the single source of truth for everything that has happened. And the hands are ephemeral sandboxes that do the actual work, with credentials held separately in a vault rather than inside the sandbox.
• 3:15
  damra: The append-only log is what makes the whole design work. It means any component can crash and the system recovers by replaying the log — the brain is stateless, the sandboxes are disposable, and the only thing that has to survive is a file you only ever append to. That is database engineering applied to agents. Write-ahead logs have kept Postgres alive through power failures for decades, and there's something a little funny about frontier agent design in 2026 rediscovering the same move.
• 3:46
  lenar: He was direct about the credential problem too. In a single-container design, the agent has unsupervised access to whatever secrets live in that container for the entire run. Twelve unsupervised hours with production credentials is a different risk than twenty minutes. Separating the vault from the hands means the sandbox gets scoped access when it needs it, instead of holding the keys all day.
• 4:09
  damra: Which matters more as these things get steered by whole organizations. He showed something called Claude Tag — a single agent instance that multiple people in an org can steer, with shared identity and organizational credentials. That's a different product than a coding assistant tied to one developer's laptop, and it only works if the credential story holds up under an audit.
• 4:33
  lenar: Then there's verification. Martin's position is that a model grading its own homework pollutes its own context — it confabulates success and then builds on the confabulation. So the harness alternates context windows: one context does the work, and a separate model call checks the output against the stated goal, and the loop repeats until the check passes. The worker and the checker never share a context.
• 4:58
  damra: That rhymes with what yesterday's episode covered from the outside — machine-checked release gates for agent harnesses. This is the vendor describing the same principle from inside: don't let the entity that did the work be the entity that certifies the work. Accounting figured this out centuries ago. [chuckle] Software is catching up.
• 5:18
  lenar: Okay, the dreaming. Long-running agents work in discrete sessions, and each new session begins with no memory of what came before, so the system keeps persistent memory stores between them. The problem is those stores accumulate errors. Martin's answer is an out-of-band process that runs after a session ends: it reviews the full trace, finds places where the agent's beliefs diverged from the record in the log, and rewrites the memory store to correct them. They call it offline dreaming, and the name fits — it runs while the agent is off, and it consolidates experience the way sleep is supposed to.
• 5:54
  damra: And his demo makes it concrete. They ran a Pokémon-playing agent that kept falling through the same trapdoor because its stored map was wrong — a mislocalized memory. Session after session, same hole. After dreaming corrected the memory, it stopped. And that's the specific image to hold on to: a persistent memory error doesn't look like a crash or a wrong answer. It looks like an agent confidently walking into the same hole every day because its notes are wrong.
• 6:23
  lenar: One line from the talk compresses his whole thesis, so let me give it to you verbatim: quote, the paradigm of loops paired with very high-capacity models is a very good general primitive for long-running asynchronous work. End quote. The loops are the harness — execute, verify, correct, and consolidate. The model supplies the capacity, and the loops supply the trustworthiness.
• 6:46
  damra: And the sharper implication is where he locates the bottleneck. His argument is that infrastructure, not intelligence, now limits production agents. The Mythos models can already run for twelve hours. Whether anyone lets them depends on checkpointing, credential scoping, and verification — the material in this talk. That's Anthropic saying the model race has a second axis, and it runs through session logs and credential vaults rather than benchmark scores.
• 7:13
  lenar: My read is that this talk gets mined the way the original ReAct paper got mined — not because any single idea is brand new, but because a lab put its production patterns on the record with names attached. Event log, verifier loop, and dreaming. People will be using that vocabulary in design docs within a month.
• 7:33
  damra: Hold on to the dreaming idea, though — because one of today's papers puts a warning label on exactly that kind of memory rewriting, and the two fit together better than you'd expect.
• 7:43
  lenar: Before the papers, the human side. The Guardian published a reported interactive by Varsha Bansal — the dateline says July twelfth, and it circulated widely in feeds last night — titled, roughly: chasing new skills, going back to basics, and pushing for collective action — how software engineers are adapting to AI. It profiles working engineers, and those three phrases in the title are the three strategies she found in the field.
• 8:10
  damra: And they're three different strategies, almost opposite ones. Chasing new skills means moving toward the wave — learning the agent tooling, the evals, and the orchestration. Going back to basics means moving under it — betting that fundamentals like systems design and debugging judgment hold their value when code generation gets cheap. And collective action means changing the terms of the game rather than changing your own position in it.
• 8:36
  lenar: One of the engineers profiled, Matt, studies on his commute — reskilling in the margins of a job he already has, which tells you something about how much slack these adaptations are happening inside. The backdrop the piece draws is that software engineering was one of the best-paying professions in the United States as recently as 2022, and the years since have brought layoffs and underemployment to a field that thought of itself as permanently scarce.
• 9:02
  damra: The collective-action strand is the one that goes beyond what this story usually covers. Engineers have historically been the least unionized well-paid profession in America, partly because individual bargaining worked so well for them. If that calculus is shifting — and this piece reports that some engineers now think it is — that's a bigger cultural change than any tool release. You organize when you stop believing you can negotiate alone.
• 9:29
  lenar: And there's a light connection to the Meta story from Tuesday — the restructuring that folded mandatory AI training-data work into engineering jobs. The adaptation pressure isn't hypothetical; it's showing up in job descriptions. What the Guardian adds is that the response is starting to look organized rather than purely individual.
• 9:48
  damra: The back-to-basics group is making the most specific technical bet: when everyone can generate code, the scarce skill becomes knowing which code is wrong. And notice that's the same bet the industry just made at the architecture level — Martin's whole verifier-loop design is the back-to-basics position, expressed in software.
• 10:08
  lenar: Okay, papers. The one I'd lead with is from Tao An at Hawaii Pacific University, titled Fidelity Before Structure. It takes direct aim at a design premise most agent memory systems share — that you should distill conversation history into structured artifacts, extracted facts and decisions and events, instead of storing the raw text. Mem0 and the MemGPT lineage — a whole product category is built on that premise, and the vendors say so explicitly in their own papers.
• 10:38
  damra: And the result inverts it. He runs a controlled ablation — same retriever, same reranker, same judge, same model, and the only variable is what gets stored: verbatim conversation chunks versus extracted typed artifacts. Verbatim chunks win by 15.9 points on the LoCoMo benchmark and by 22 points on LongMemEval. The extraction pipeline doesn't even beat naive retrieval-augmented generation overall. Within one fixed pipeline, the distillation everyone assumed was a feature turns out to be the cost.
• 11:12
  lenar: His opening example is the best summary of the mechanism. In turn three of a fifty-turn conversation, a user says: please use type hints everywhere. Summarization stores that as, the user prefers type hints — which sounds fine and has silently deleted the word everywhere. On constraint questions like that, his probe finds summarized memory answering at 14 percent exact match versus 91 percent when the verbatim wording is retrieved. A 77-point gap between compression and fidelity, from one deleted quantifier at a time.
• 11:47
  damra: He names the mechanism lossy distillation, and he's precise about scope: the culprit is replacement, not structure per se. Adding artifacts alongside verbatim chunks preserves accuracy; substituting them forfeits the gap. He also concedes the one place chunks lose — they abstain worse, meaning they're more likely to confidently answer when they shouldn't. And he ran six confound controls plus a stronger-extractor swap to close off the obvious objections, so this isn't a strawman extractor getting beaten up.
• 12:19
  lenar: He even did the cost math, because the standard defense of extraction is efficiency. Verbatim chunks cost more per query but less per correct answer — twelve and a half dollars per thousand correct answers versus about fifteen for the artifact pipeline. So extraction loses on accuracy and on cost per unit of usefulness, which removes its last line of defense.
• 12:42
  damra: And here's the tension with the Anthropic talk. Offline dreaming rewrites memory stores — a model reads a trace and updates the notes, which is exactly the kind of lossy step this paper warns about. The saving grace in Martin's design is the append-only log: the verbatim record never goes away, so dreaming can always be re-grounded against it. Build the dreaming without the log and this paper says you'll pay for it in silently corrupted memories. Keep the transcript. Annotate it, and don't replace it.
• 13:14
  lenar: If you're building agent memory this year, that's the sentence to keep: fidelity first, structure second, and never delete the source. The code and data are on GitHub under cog-canvas, so the whole ablation is checkable.
• 13:28
  lenar: Two more from the same arXiv batch, at a faster pace — and a note that these are version-two updates of recent papers, not first postings. First, NexForge, a framework for synthesizing agent training data. The pitch is requirement-driven task synthesis: instead of generating training tasks from whatever tools and repositories you happen to have — which bakes your infrastructure's biases into the data — it starts from high-level capability requirements, profiles real-world demand into scenarios, and then automatically constructs the repos, dependencies, and runtime configurations to make each task actually executable.
• 14:08
  damra: The numbers are the eyebrow-raiser. The same pipeline generated thirty-six hundred terminal tasks and two thousand office tasks with no domain-specific infrastructure, and fine-tuning on those took a Qwen3.5 base model from 22.5 percent to 52 percent on Terminal-Bench 2.0, and from 813 to 1338 Elo on GDPval. Scaling to 43 thousand terminal tasks pushed it to 58.4 percent — which the authors claim surpasses Claude Opus 4.6. Their Nex-N2 model family reports 75.3 percent on Terminal-Bench 2.1 and 1585 Elo, which they describe as state-of-the-art for open source.
• 14:56
  lenar: And the skepticism is standard but necessary: these are self-reported numbers, and synthetic tasks and the eval can end up sharing a distribution by construction. The frontier-comparison claim needs independent replication before it means much. What I do buy is the direction — that task synthesis, not model weights, is where open-source agent training is compounding right now. The models are released, so the replication can actually happen.
• 15:23
  damra: The Alipay benchmark is the more grounded artifact of the two. Alipay-PIBench evaluates coding agents on real payment integration — nine product-specific projects, eighteen task instances, and it's the advanced tier that makes it interesting: notification idempotency, abnormal-transaction handling, refund safeguards, and fund-safety controls. It's testing whether the money stays where it should when the webhook fires twice, not just whether the code runs.
• 15:53
  lenar: Their motivating point is that an agent can make payment code look runnable while trusting client-side payment status or skipping signature verification, and no general-purpose benchmark would catch it, because the code compiles and the happy path works. Across the six models they tested, giving agents packaged domain skills improved mean pass rates by about ten percentage points, with per-model scores ranging from roughly 69 to 91 percent.
• 16:22
  damra: That ten-point gap is the number practitioners will remember: domain knowledge, packaged as skills, moves agent performance by ten points on tasks where the downside is losing actual money. Evals are drifting toward the places where mistakes have denominations, and that's the right direction for them to drift. There was a fourth paper in the batch too — diversity collapse in self-play training, with vocabulary dropout as the countermeasure — one for the people building their own curricula.
• 16:52
  lenar: Last segment — two robotics papers, briefly, because they've arrived at problems this episode has already covered twice from other directions. The first is Zero2Skill, from GigaAI and a long list of university collaborators. It's an autonomous data-collection system for robot manipulation where the human operator's corrections persist. The robot collects demonstrations on its own. When a phase keeps failing verification past a retry budget, it pauses and asks a remote operator, who answers in plain language. A language model parses that correction into a structured adjustment and stores it in what they call a Corrective Memory, consulted on every subsequent round.
• 17:34
  damra: And the whole point is that a correction issued once keeps working. In their desktop-clearing testbed, that cut human working time to 16 percent of full teleoperation while matching its collection success rate, and language corrections raised average single-attempt success from 12.5 to 47.5 percent. It's the same insight as Anthropic's dreaming, reached independently on physical hardware: the expensive resource is human attention, and memory that persists across sessions is how you stop spending it on the same problem twice.
• 18:08
  lenar: The second is RouterVLA, an update on routing across pools of vision-language-action policies — the models that drive robot arms. The observation is that robotics teams maintain several policies but deploy one global winner, and the pool has conditional structure a single winner can't express — one policy dominates familiar scenes, another dominates a specific perturbation. Their system reuses smoke-test evidence and a small probe budget to pick the right specialist per condition, and to decide which new candidates even deserve admission to the pool.
• 18:44
  damra: Modest gains in this version — around sixty percent held-out success, a point and a half over a semantic shortlist — but the problem statement is what transfers. Anyone running multiple coding agents is doing this by hand right now: which model for which task, and when does a new checkpoint earn a slot in the rotation. The robotics community is just writing the admission rules down first.
• 19:08
  lenar: So today came down to one principle reached from three separate directions: keep the verbatim record, verify with a separate pair of eyes, and let corrections accumulate instead of evaporating. A lab explained how it keeps a twelve-hour agent from wrecking its own run, and the engineers underneath it all are deciding whether to ride the wave, duck under it, or organize against its terms.
• 19:31
  damra: And the near-term test of the Martin talk is simple: someone ships an open-source imitation of the event-log-plus-dreaming pattern, probably within weeks, and then we find out whether the pattern or the models were doing the heavy lifting.
• 19:45
  lenar: If it shows up, we'll read the code on the air. I'm Lenar Kess, and that was Braid for Thursday, July twenty-third.