Latest / TeqTalk with Jas / AI Harness: What Makes Agents Reliable, Not Just Smart
Transcript
- Teqfocus: Welcome to Tech Talk, where we break down the architectures shaping enterprise AI. Today, a term you're going to hear a lot more as AI has moved out of chatbots and into the systems that actually do things. And the term is harness engineering. Here is a scenario two engineering teams, same frontier model, similar APIs, similar databases, same problem to solve. Team A's agent investigates, picks the right tools, remembers What it actually did recovers when something breaks and finishes the task. Team B's agent starts fine, they loses context halfway through, calls the wrong tool, retries an action that already succeeded, and it either gives up too early or loops forever. Same underlying intelligence, widely different reliability. So what made the difference? Increasingly, it's not the model. It's the engineering system built around the model. Think of the model as a reasoning engine. The harness is everything around that engine, deciding what information reaches it, what actions it is allowed to take, what gets remembered, and what happens when something goes wrong. So by the end of this video, you'll have a checklist you can run on any AI agent, including the ones you're deploying it right now. But first, why do we suddenly need one? Because three years ago we didn't. Let's see the history. Stage one, prompt engineering twenty twenty two. November twenty twenty two, Chat GPT kicks this off. You type a request, the model answers. For summarizing, classifying, drafting, generally very great. But notice what's not happening. It isn't touching your database or sending an email. It can describe an action beautifully, it just cannot take one. It changes the text on your screen, not the world. So the industry asked. What if we let it? Then came stage two, which was retrieval and tool usage. We wired models into external systems. And then RAG went mainstream once Langchain arrived in late 2022. Then OpenAI shipped function calling. The model could formally say, call this API with these exact parameters. So that was real unlock. But every call was one off. One action, one result, done, no loop. The model couldn't look at what came back and keep going. Which brings the next question. What happens when a task takes more than one step and nobody is reprompting it every time? That was the stage when agents came into the picture. The research idea is React. That was published at ICLR 2023, interleaving reasoning with actions and observations instead of treating them separately. Auto GPT went viral in spring 2023 and the infrastructure caught up. Anthropics model context protocol in November 2024 and by 2025 the real production tooling, OpenAIs, Agent SDK, Anthropics, Claude Code, we're just booming up. When it works, it's powerful. But that loop opens a whole category of failure. Who controls how long it runs? What if it forgets what it did three steps ago? What stops it taking the same destructive action twice? Hold on to this one because we are going to take an example and run through it in the whole episode. A support agent handling refunds. It processes one, everything looks fine, but the confirmation gets lost on the network. And to the agent, that looks like there is a failure. So it tries again. Now your customer has been refunded twice. And nobody ever told the agent that was possible. The model didn't do anything wrong. It did exactly what it was asked to. The system around it wasn't built to catch that. And that's the shift. The moment a model can observe, act, and decide what happens next on its own, you're not answering just a question. You are running a whole live system. And a live system doesn't need just a smart participant, it needs an operator. There comes our stage four harness engineering. That gap, a powerful loop with nobody operating it, is what the state solves. The fix isn't a smarter model, it is the operating system around that model, like how I've mentioned before. And that's the harness. The term hit the mainstream in Feb 2026 when OpenAI published a five month experiment. A real product, zero manual written quote. We'll come back to those numbers in a while. So what is harness mechanically? The model gives us intelligence, the harness gives that intelligence an operating system. Together they form an agent. And when I say harness, I mean everything that isn't the model. own reasoning, how it executes tools, what state it tracks, what it is permitted to do. Here is no industry standard here. Nobody has published a spec so far, and for this episode I'm organizing it around nine layers. Rather than throw nine abstractions at you, let's carry on that example and go through those nine layers. So back to that refund agent. It touches money, it's a multi-step. And it can genuinely hurt a customer if it gets wrong. Sounds like one instruction, but it's entire runtime. Let's talk about it. The first layer is context. First question before the model generates a single token is what does it say? The tempting answer is everything. Full account history, every pass ticket, three years of transactions. More information, the better decision, right? That's not right. Context is a finite resource. Load all that and the thing that matters. This disputed charge gets buried. Attention gets diluted, cost can go up and accuracy often can go down. So the harness decides. Load the disputed transaction, the last few interactions, the policy rules that apply, everything else stays retrievable but not preloaded. And the rule for this is Give the agent the smallest amount of high signal information it needs for the next good decision. OpenAI reads the same principle from the opposite direction. So instead of giant instruction manual, they treat their agent instruction file as the table of content. Roughly a hundred lines pointing deeper sources somewhere else. But that's only in the opening context. And here is a distinction that catches teams out. What do you load at the start is context delivery, and what happens over the next two hours as that window fills in is context management. Because our agent doesn't stop after one exchange. It pulls the transaction, it reads a policy, checks history, and all that lands in the window until the original instruction is competing with. 40 screens of retrieved data. So the harness needs a second policy. What gets compacted? What gets summarized? What gets dropped? Get the first thing right and the second wrong, and your agent is sharp for 10 minutes and vague by 40 minutes. So tools here, Langchain or Lama Index for retrieval pipeline, a vector store like Pinecone or PG Vector and MCP for Exposing context in a standard way. So our agent knows what it needs to know. But knowing and deciding what to do are different things. And who is running those things? Let's look into our layer two, which is orchestration. Our agent decides to pull up the original transaction, but the model doesn't execute that. It requests it. Something else pulls a record, hands back the result, and then asks, call the model again. Is this done? Does a human step in? That's orchestration. And here was the rule that matters most. Use model intelligence for ambiguity and deterministic software for invariance. The model decides whether the customer's story is credible, genuinely ambiguous, good use of intelligence, but a hard cap on retries. That's not a judgment call, it's a fixed number in your application logic. Let the model set its own retry limit and you have built something that can loop. In forever. Lang graph is the most used framework here. It models the agent as an explicit state graph. Which is that rule that makes concrete? Temporal for genuinely long-running work. So far, we have assumed one agent doing everything. There's a point where that stops being the right shape. That brings to our third layer, which is sub-agents. Our refund agents does two different things. It investigates, which means it reads a dispute. Pulse history weighs whether the claim is credible and it executes. That means it validates, authorizes, issues of refund. Those want different contexts. Investigation wants breadth, execution wants precision. So you split them. A research sub agent that investigates and returns a recommendation. A refund sub agent that executes it. So each runs with narrow context. Each gets only the permissions its job needs. The research agent never holds a refund authority. Two shapes manager and specialists where a lead agent delegates out and handoff where control passes fully from one to another. But here is a trap. Some agents are not free. Every split adds a boundary and boundary lose information. You have also multiplied token cost, latency, and places where Buck can hide. So the honest rule is do not spend. Split until one agent is visibly failing, or actually there is a need of the second agent. Crew AI and Microsoft's Origin are the known multi-agent frameworks. Langgraph handles it as subgraphs. So our sub agents have decided the refund is valid. One wants to issue it, there is where everything changes. The model produces a structured request, a refund call with a customer ID, transaction ID, amount, reason. And here is a single most important idea in this episode. The model saying refund this transaction must not automatically move money. It must be checked whether this is a real transaction or does it belong to that customer? Is the amount the actual charge or did the model hallucinate a number? And critically, has this transaction already been refunded? The last check is the whole ballgame. Remember our failure. Refund succeeds. Network drops, agent sees an error, agent retries, and without a check, your customer gets refunded twice on the charge. The property that prevents it is idempotency, which is doing something twice has the same effect as doing it once. And notice none of that is AI engineering. Validation, authorization, timeout, retry policy, distributed system discipline, these are all existing from decades. AI didn't make traditional software engineering obsolete. It makes it more and more important because now a probabilistic system initiates the action. So that described as the layer four, which is tools that turn your intent into actions. Now coming to the next layer, which is skills, our agent can issue a refund. That's a capability. What it doesn't have is your company's actual dispute procedure. Because at your companies there is a real sequence. Check whether the charge was duplicated before you check whether it's a fraudulent. If the customer has disputed twice this quarter, flag it rather than your order approved. Nobody derives that from first principle. It is institutional knowledge and right now it lives in a conference page three people have read. That's a skill. A reusable procedure written down that agent loads when it is relevant. And It isn't just context or tool. A tool is capability that is issue a refund. A skill is a procedure that uses capabilities. How we can handle the dispute here? In what order? With what exceptions? And unlike context, this isn't preloaded into every prompt. It sits in a library and gets pulled when the task matches. And here is why this layer matters commercially. Every other layer is infrastructure. Broadly the same at your company. As at your competitors. So this is where your specific institutional knowledge gets encoded. Your vector database is into mode. 20 procedures that encode how your business actually operates might be. Entropics agent skill is the most explicit implementation of this. In a lot of production systems, it is simply a version folder of markdown files. Unglamorous but extremely effective. So our agent knows what to do and how we do it. Now What does it remember when the process breaks? Which brings us to our sixth layer that is state and memory. Two different problems people constantly conflate. Operational state, which is where we are right now, with step, which tool ran, was a refund issued, was approval granted, and knowledge memory, which is what happened before. What should shape the future decisions? This customer has disputed several charges recently. Context for a judgmental call, not a record of this run. You need both. Scale it and it is obvious. Say your agent isn't handling one refund, but a batch of 50. And it crashes on number 37. Conversation history won't save you. You need to know exactly which refund committed, which didn't, and where it is safe to resume. That belongs in durable external state. The rule of thumb is if forgetting something could cause an irreversible action to happen twice, do not store it only in model context. Persist it. Keep the two separate in your tooling too. For operational state, Langgraph, checkpointers, temporal, or honestly, just postgray, which is what most production systems use. For knowledge memory, there is a new category Zep, Litter. And the simplest version, straight from OpenAI's experiment, a progress file committed to Git, which is completely durable. So that's what our agent knows, decides, and remembers. Now a different question. Not what it can do, but what it is allowed to. That makes us move to our seventh layer, which is control permission and sandboxing. Our agent has a technical ability to move money. Should it always be allowed to? Least privilege. Only the permissions the job needs. Rs read transactions and issues refunds. It doesn't need to delete accounts, modify price, or export to your customer database. If it can, that's a breach waiting to happen. In approved boundaries, some actions run autonomously. Some pause. A small refund goes through automatically. Above a threshold you said it stops and waits for a human. And sandboxing, that is the execution environment question. Contain the blast radius. The agent runs inside a bounded environment. So even when something goes wrong badly, the damage has a ceiling. Docker, Gvisure, or hosted sandboxes like E2B for isolation. Open policy agents for policy as code, it applies to agents exactly like it applies to microservices. Now here is a nuance that makes the slate generally very hard, and I want you to hold on to it because human approval is not automatically safe. We are coming back to that with real data and I think it'll change how you design this layer. But before we get there, none of this matters if you could see what happened or prove it worked. That brings us to the layer, which is the eighth layer, and that is verification and observability. I am treating this layer as one because they are the same discipline pointed to two questions. Observability asks, can we reconstruct what happened? And verification asks, was it actually right? And you can't answer the second one without the first. So that sequence is a trajectory. The full path that produced the outcome, not just the final answer. When a double refund lands on your desk tomorrow, the agent made a mistake, is useless. Did it pull the slate data? Did it call the refund twice? Or did the tool fire twice on one call? Was approval granted or quietly bypassed? Which sub-agent made the call? Now the verification half of a customer. Our agent reports refund process successfully. The customer has been credited. Very confident, well formatted, plausible, and potentially completely false. Because the question isn't, did the agent produce a good sounding answer? It is, is the money actually back in that customer's account? That's the difference between checking the output and checking the outcome. The real state of the world. And that is a query against your payment system, not a judgment about a sentence. And even that is not enough because an agent can reach the right outcome down an unacceptable path, issuing the correct refund, but pulling out four other customers' records to get there. So you verify on your several axis was the output correct? Did the real world state change? Was the plan acceptable? And was the cost reasonable? Langsmith, Langfuse, Brain Trust all traced and evaluate. But the structurally important point is OpenTelemetry now has semantic conventions for Gen AI, meaning agent traces live in the same data dog you already use. You are extending the observability stack, your SREs already watch, not standing up a parallel one. So we can see what happened and prove whether it worked, which leaves the last piece. What do you do when it didn't? And that is a recovery layer. Back to our batch of 50 refunds crashing on 37. Five second task, just restart. But you cannot restart this one blind. Refunds 1 through 36 go out a second time. So recovery means answer three things. What already committed, what is safe to repeat, and where do we resume? And this has to classify failures. A network timeout, retry, it's a transient. Invalid credentials, a policy violation, and it has to classify failures. A network timeout, retry, it's a transient. Invalid credentials, do not. That's not fixing itself. A policy violation, stop and escalate. So checkpointing, durable state, retry policy, classic example, engineering. The only new ingredient in this is a probabilistic model now sits in the middle of it. This is a layer where AI native tooling is thinnest and boring infrastructure wins. Temporal is strongest answer. Durable execution means the workflows arise, a crash, and resumes exactly where it stopped. Which is precisely the step 37 problem. So look at what we built. One sentence, investigate this dispute and refund it if valid. Turned into nine engineering layer. The model supplied the judgment. The harness made it safe enough to trust with our customers' money. Now let's see OpenAI's five-month experimentation. A real internal beta product, zero manually written quote, every line of application logic, test, CI config, documentation, tooling generated by codex, empty repository, late August 2025. Five months later. On the order of a million lines. But here is the number that stopped me. Roughly 1500 pull requests open and merged, driven by three engineers. About three and a half merged PRs per engineer per day. And the part OpenAI flagged as surprising as a new team grew to seven, throughput per engineer didn't drop, it went up. Their estimate. Roughly one tenth the time manual coding would have taken. But lesson isn't the line count, it is what changed about the engineer's job. OpenAI is framing the primary work shifted from writing code to designing environment, specifying intent, and buying feedback loops. And listen to how they describe failure because it's almost a definition of that discipline. When something fails, the fix was almost never try harder. Engineers stepped in and asked, What capability is missing and how do we make it legible and enforceable for the agents? That is the context tool and skill at scale in production. Now that proves out the layers you can see easily. But remember the one I asked you to hold that human approval isn't automatically safe. Here is what that gets paid off with data published this month. Enthropic hit exactly this problem inside its own coding agent, Claude Code. It shipped with a Conservative default. And on paper, it is a design most of us would call responsible. Ask for human approval before every file is written or every shell command, every network call. And then they looked at the telemetry. In their March write-up, users were accepting about 93% of the prompts. By their August post, it was figured out to be 97%. And their own framing was blunt. An approval rate that high suggests. People are clicking through reflexively rather than reviewing each command. The pauses were happening, the reviewing wasn't. That's approval fatigue. And that is the detail I would like to point any architect to. Same users, same product, two kinds of dialogue. When Claude presents a plan for approval, users reject 39% of them. When it presents an individual permission request, the rejection rate is 3%. People read Plans. People do not read the 11th permission prompt of a session. That gap is the evidence it's generally fatigue. Not everything approved was safe. So in March 2026, Anthropic shipped automode, a classifier that reviews each action before it runs and only escalates what something looks irreversible, destructive, or aimed outside your environment. And then they did the thing which I wish most vendors did. They tested whether it was actually better. A controller study, over 1000 paid tester, partways through each session, researchers swapped a routine permission prompt for a generally dangerous command. Nothing dangerous ever ran, only the text change. Human reviewers caught it 13.6% of the time. That's a 143 out of 1053. Auto mode blocked 89% of the same commands. And the fatigue curve is right there in that data. Early in the session, human blocked out 17% of dangerous commands after 50 or more prior prompts that dropped to about 5%, and the classifier's block rate stayed flat regardless of the session length. On August 14, 2026, Automode became the default for Pro Max and Team plans. Now bring that back to our refund agent. If you designed it. So a human approves every refund above a threshold. You probably felt good about that. This data says go check whether those approvals are real reviews or just clicks. And notice the shape of the fix. Anthropics didn't remove the human. They moved the human's job from clicking prompts to defining the boundaries before the agent ever starts. Their own documentation still warns that auto mode doesn't guarantee safe. And recommends human review for sensitive operations. So the lesson for the control layer. Add a human checkpoint is not the same thing as add safety. A checkpoint only works if something is actually looking and if they are not, what you need is containment, sandboxes, scope permission, boundaries set up front, and no more clicking. So one agent through nine layers, two companies living through them. Let's turn that out into something you can use tomorrow. Next time somebody shows you an enterprise AI agent, run these. First is context. What does it know right now and what happens as the window fills? Orchestration. Who decides what happens next? Subagents. Is this split for a reason or for a diagram? Tools. What can it actually do? Skills. Does it know we do this? Or is it guessing? State. What information survives a crash? Control, what runs autonomously, and who reviews the rest? Verification. Can we reconstruct the run and prove the outcome? Recovery. What happens when step 37 fails? That checklist is really shorthand for one bigger idea, which is where I want to leave you. Three distinctions: prompt engineering asks, how should we talk about model? Context engineering asks. What should the model know right now? And how does engineering us? How do we engineer the operating environment so intelligence turns into reliable actions? And here is what I would like to take the history in this episode. This isn't a settled playbook. The term is barely six months old. Both case studies were looked at were published this year, one in Feb, another one in this month, August. The people further Ahead are still publishing their own telemetry, still testing assumptions against controlled studies, still changing defaults when the data tells them to. This is not a weakness in the discipline, that's a discipline working. Measure what your system actually does, not what you design it to do. And publish the results. Because intelligence can propose. The hardness in the human designing it determine what actually happens. This is Tech Talk. If you found this useful, subscribe. We go beyond the headlines and break down the architectures that actually matter when these system hit production.