Planning, Verification and Building Your Own (Harness Design, Part 3)

Deep End Era. Engineering with Sankalp · Oct 12, 2026 · 50m

Ines's agent now either gives up too early or keeps going after it's done. Fan, Zhang et al.'s planning ablation explains both. A persistent to-do list added 11.6 points for Nemotron-3 30B (median run 5 turns without it, 40 with it). It cut SWE-Bench cost by about 30% for Nemotron-3 550B and 32% for Mistral Medium 3.5 by trimming redundant verification, with a small, non-significant drop in accuracy. The episode covers verification and tests that give wrong answers, SWE-Bench Verified contamination and the SWE-Bench Pro retraction, the harness as a hidden variable in leaderboards, latency and…