Latest / Embodied AI 101 / Test-Time Compute Scaling for Robot Policies (DIRECT)

Transcript

Test-Time Compute Scaling for Robot Policies (DIRECT) · Embodied AI 101

• Test-Time Compute Scaling for Robot Policies (DIRECT). Modern robotics increasingly uses vision-language models (VLMs) as high-level planners, generating sequences of sub-tasks from an image and an instruction, which a lower-level controller then executes. These VLM planners can be powerful – they can “think” through complex instructions or recall long action histories – but at a cost. Using chain-of-thought reasoning or a much larger model can slow inference to tens of seconds per plan, and pulling extra context (memory) adds even more latency and compute. In the new Stanford-led work DIRECT, Jadelynn Dao et al. scrutinize exactly this tradeoff: which compute levers (model size, reasoning depth, memory length, etc.) actually pay off on which tasks, and how can a system allocate inference-time compute efficiently? In short, they show that naively always using the beefiest planner is wasteful, since many tasks are solved just as well by a faster-but-cheaper planner. Their solution is a lightweight learned router that reads the scene and instruction, then dispatches each query to the smallest capable planner. Remarkably, this strategy often matches or even exceeds the success of an “expensive” planner while cutting latency by up to 65%. The Compute–Performance Tradeoff in Robot Planning. In a typical hierarchical robot system, a VLM reads the current visual scene and a high-level instruction, and outputs a goal or a sequence of subtasks (e.g. “pick up the red cube and then place it atop the green box”). A lower-level controller then executes these subtasks. Over the last few years, researchers have found that increasing inference-time compute for such VLM planners can improve their capabilities: for instance, one can run more chain-of-thought steps, use a larger VLM, or maintain a longer history of observations. Indeed, recent works like Google’s Gemini Robotics 1.5 flaunt multi-stage thinking (“why” loops) and memory recall, and the Do as I Can, Not as I Say line has shown benefits of deliberate reasoning for affordances. However, each extra “think” or extra token has a real-time cost. Dao et al. point out that we often pay steep latency for small gains: e.g. on a real Franka robot test, a “thinking” VLM was 3.8× slower than a fast (non-thinking) variant yet produced exactly the same success rate. That means most of the time the extra deliberation was wasted. In other words, doubling or tripling compute at inference does not uniformly boost performance – it only helps with certain hard tasks, while on many “easy” problems a simple planner would suffice. Inspired by this, the authors advocate a different philosophy: treat the choice of planner as dynamic, not fixed. In recent LLM work (e.g. FrugalGPT, RouteLLM) the idea of “model routing” has been explored – queries get sent to the smallest model that can handle them, balancing cost and quality. Dao et al. apply this to vision-language robot planning, stressing that here the decision must consider visual context, not just text. In their words: “effective allocation requires routing policies that infer task demands from scene and instruction context”. The top-level question becomes: when and where should we invest inference-time compute for a robot task? Is a light-weight model enough, or do we need deep reasoning or extra memory for this particular instruction and scene?. When Extra Compute Helps (and When It Doesn’t). To answer that, the authors first perform a diagnostic analysis along three axes of compute-scaling: (1) Reasoning depth (chain-of-thought steps vs direct output), (2) Model size (small vs large VLM), and (3) Memory context (short history vs rich context). They evaluate on benchmarks like VLABench (long-horizon language-conditioned manipulation tasks) and RoboMME (memory-focused tasks), and also on a real Franka robot (the “DROID” setup). The results reveal striking patterns:. Chain-of-Thought vs Single-Step: Many VLM planners have a “fast” mode (just answer directly) and a “thinking” mode (internally break down reasoning). The authors compare Qwen3-VL 8B in Instruct mode versus its Thinking mode on VLABench. As expected, Thinking improves overall, but only on some problems. Concretely, Dao et al. found that “while thinking improves average performance, on 44% of tasks the Instruct model matches or outperforms the Thinking model, with orders-of-magnitude faster inference”. In practice this means nearly half the tasks saw no gain from the slow reasoning process. The cost difference was enormous: the Thinking planner took on the order of 20+ seconds, whereas the Instruct planner finished plans in under a second. This suggests that for many relatively simple instructions (e.g. “bring me the red block”), heavy “thinking” is unnecessary. They confirmed this behavior on a physical robot too. Across five pick-and-place test tasks (categorized by whether they need common-sense, spatial, or perceptual reasoning), the non-thinking planner got the job done on two tasks but failed on three, where Thinking succeeded. A smart router would thus rely on the cheap planner for the easy cases and escalate only for the others. Indeed, Direct’s learned policy did exactly that: it “escales to the thinking model only where it helps, recovering the Thinking variant’s success at a significantly lower latency”. In fact, Direct on the robot got near-oracle success (essentially matching Asking the “thinking” model exactly when needed) with far less waiting time. Even more dramatic is a multi-step task in hardware: a grocery-bagging scenario where the robot must bag items from heaviest to lightest, planning step-by-step with new observations each time. The authors compared three strategies: always using the 9B Instruct planner, always a 9B Thinking planner, or Direct (which re-chooses at each step). Here the Thinking planner succeeded 90.5% of the time but was slow (~19.6s), while Instruct got only 47.6% success (and was fast, ~2.2s). Strikingly, Direct achieved 95.2% success at 6.85s latency – beating Thinking in success and running nearly 3× faster. In summary, chain-of-thought depth is only beneficial for specific cases; many steps can be handled without it, and a learned router can exploit that. Model Size: Intuition might say “bigger is better”, but the study finds a nuanced picture. The team tested Qwen3-VL Instruct models from 2B up to 235B parameters. On VLABench, success did not steadily climb with size; in fact, performance curves were irregular. Surprisingly, smaller models sometimes ran slower than larger ones, simply because small models can be more verbose! The paper notes that “since verbosity is not tied to parameter count, smaller variants sometimes generate too many tokens and run slower than larger ones”. The real advantage of larger models was in capability breadth: the biggest models could handle certain complex skills that small ones failed. But those gains only appear on tasks that demand those new skills. On many routine tasks, the 2B or 4B model is enough. Thus, giving every task the huge model wastes compute. Direct evaluated routing on model size by building pools of models (e.g. [2B,4B,8B,32B]). With a router, they saw performance jump monotonically as more models were available. For instance, a router over {2,4,8,32B} steadily improved success at each step, whereas using the 8B or 32B alone had dips. In one comparison on VLABench, the cumulative router (2+4+8+32B) scored 5.1% higher success and cut latency by 32.4 seconds on average, versus always using the 32B model. In effect, switching to the smaller model on easy tasks and saving the big model for the hard ones gave strictly better efficiency. The authors show that direct routing converts a jagged “bigger = more success” curve into a smooth improvement: it “recovers each model’s distinct skill coverage while discarding its weaknesses”. In plain terms, the router gets you almost all the lift of a 32B model but at a fraction of the time costs. Memory and Context: Some tasks require remembering past subgoals or observations. Dao et al. benchmark this with tasks in RoboMME, where a model can compress history in different ways. For example, one strategy is just sample fewer frames (‘FrameSamp’), or drop some tokens (‘TokenDrop’), or generate a simple text summary of past goals (‘SimpleSG’), vs. a heavy approach like MemER which retrieves past keyframes. They found again diversity of needs. On easy tasks (short recall needed), the simplest FrameSamp approach dominated – it solved nearly everything with minimal compute. Only on the hardest tasks (long recall needed) did MemER or grounded sub-goal summaries pull ahead. Quoting the paper: “On easy tasks needing short recall, the lightweight FrameSamp outperforms MemER at an order-of-magnitude fewer FLOPs; as tasks demand distant history, MemER and GroundSG have higher success.”. Crucially, no single memory scheme was best for all difficulty levels. Accordingly, Direct’s routing again wins. In simulation on RoboMME, the routed frontier (success vs cost) lies above any standalone memory model. In the authors’ words, for each difficulty suite “Direct generally achieves a success–cost point better than the most performant architecture, at a fraction of the most expensive specialist’s cost”. In other words, mixing strategies per task strictly dominates any one-size-fits-all memory scheme. On hardware, they set up two variants for comparison: a “no memory” planner (only last frame) versus a “full memory” planner (with 40-frame history + task summary). They ran tasks where the goal was either observable now or hidden until remembered. The memory model succeeded on hidden-goals and (typically) failed or was unnecessary on visible-goals. Direct’s router quickly learned this pattern: it invoked the memory-augmented model only when needed. As reported, Direct “recovers near-oracle success rate while operating at substantially lower average frames processed than the full memory-augmented planner”. It beats the no-memory planner on the hidden-goal tasks and beats the simple planner on the visible-goal tasks. The punchline: adding recall overhead on every task is needless; instead, the router loads memory only for those tasks that truly need it. Overall, these probes show a common theme: extra compute has highly task-specific value. Tasks that hinge on difficult spatial or semantic reasoning benefit from deep chain-of-thought; tasks requiring diverse skills need a bigger model; tasks with long-horizon goals require memory. But easy instances of each category can skip the heavy machinery. Any static planner (fixed size, no matter chain depth) fixes a single point on the quality–latency curve. Dao et al. summarize their insight: “large capability gaps separate cheap and expensive configurations… lightweight models suffice on many tasks… a static model fixes a single latency–capability compromise”. These observations directly motivate their solution: training a controller to infer each task’s demands from the input and allocate compute on a per-case basis. DIRECT: A Dynamic Inference Router. Building on this diagnosis, the paper introduces DIRECT (Dynamic Inference Router for Embodied Compute Tradeoffs) – a learned policy that picks, for each new task, which planner variant to use. Concretely, the system has a pool of candidate planners (M={m_1,\dots,m_K}). Each candidate (m_k) is a VLM-based planner configuration (for example: Qwen3-VL 8B without CoT, or Qwen3-VL 27B with CoT, or a certain memory architecture, etc.). When a task arrives (an image (I) plus language (\ell)), we compute a feature vector (\phi(I,\ell)) and run a cheap routing model (r(\phi)) that outputs an index (k). The robot then invokes planner (m_k) to produce the subtask plan. Because the router itself is very lightweight, its computation (tens of milliseconds) is negligible compared to the seconds of actual planning. In the experiments, Dao et al. use straightforward multimodal embeddings: they encode the image with a frozen SigLIP-family vision encoder and the text with a frozen BGE-M3 language encoder, then concatenate these vectors. On top of this embedding, they train various simple classifiers/regressors (linear models, small MLPs, k-NN, etc.) to predict the best planner. This choice worked well: the paper reports a negligible 20–50ms router time on average, in contrast to >1s for any VLM planner. During training, for each task (x_i=(I_i,\ell_i)) and each candidate planner (m_k), they precompute two numbers: a quality score (q_{i,k}) (e.g. success or progress if planner (k) is used) and a cost (c_{i,k}) (e.g. latency or FLOPs). Then they pose routing as selecting the planner that maximizes a utility (U(q,c)). For instance, one can imagine giving high reward to high success and penalizing high cost. In practice they tried a couple of heads: for example regressing a predicted success (\hat q) and cost (\hat c) and picking the model with highest (\hat q) while (\hat c) is within a bound. Their experiments show that directly regressing quality and cost signals gave the best results. To evaluate router performance, the authors use a combined efficiency metric (\eta) that balances normalized success and cost (inspired by prior model-routing benchmarks). They compare DIRECT against sensible baselines: always using the cheapest planner (“Cheap”) or always the most expensive (“Expensive”), random selection, and an outlier-detection scheme (where highly out-of-distribution queries are routed to bigger models). They also report an unattainable “Oracle” that picks the true best planner per task (using the collected (q,c) data) to set an upper bound. In every regime, the learned Direct router achieves the best tradeoff, closely approaching the oracle frontier (the gap to oracle indicates possible headroom for future). Results: Smarter Compute at Work. The bottom line of the experiments is clear: Direct yields near state-of-the-art success rates at a fraction of the cost of a monolithic big planner. For the chain-of-thought axis on VLABench, the learned router managed to recover essentially the full performance of the expensive Thinking model while cutting its inference time by about 30% on average. As the authors report, on homogeneous model pairs (cheap vs expensive version of the same family), “the cheap-to-expensive quality gap is large, and Direct recovers near-expensive quality while cutting latency by around 30%” with efficiency (\eta\approx 75%). Even when both base models were of fixed size (so only reasoning depth differed), Direct beat both the pure cheap and pure expensive approaches: it not only trimmed latency but actually achieved higher success than either static choice, because it combined their complementary strengths. In mixed-family pools (e.g. routing between different model architectures), the gains remained robust: Direct got most of the expensive model’s edge at ~30–40% lower latency. The hardware tests mirror these gains. On the real-robot single-step tasks (Fig.6 in the paper), the non-thinking planner was very fast (≈0.9s) but missed some tasks, while the Thinking version had high success but was ~20s. Direct’s policy essentially hit the oracle: it succeeded on those very tasks that needed thinking, and skipped it otherwise. The paper notes that Direct “achieves close to the Oracle on both success and latency, and outperforms both OOD- and random-based routing”. In practical terms this means the robot finished tasks correctly almost as much as always using the complex planner, but with only a fraction of the wait time. The grocery-bagging example underscore this: Direct finished most runs successfully (95.2% success) in 6.85s (average), versus the Thinking model’s 90.5% success and 19.6s. On model size, the router turned the irregular scaling into monotonic improvement. With a pool {2B,4B,8B,32B}, their cumulative routed solution outperformed any single model: it kept the best of smaller models for easy cases and fell back on the large one only when needed. By the time they allowed up to a 32B planner, Direct’s solution had 5.1% higher success than fixed-32B and was 32s faster on average. (Put another way, using the router versus always the 32B model saved more than half a minute of inference per task!) Across all the tests, Direct consistently achieved the maximum “success–cost Pareto frontier” of the pool. The paper’s aggregated results show that on both VLABench and RoboMME, Direct outperforms all baselines: it beats the strongest single model at substantially lower cost. Finally in the memory domain, Direct again wins. The plotted success–cost curves (Fig.9) illustrate that, for each difficulty tier, the routed frontier sits above and to the left of the best fixed strategy. In other words, there was no single memory architecture that could match the router – Direct left each specialist behind by cutting its cost and matching or exceeding success. On the robot, Direct essentially achieved oracle performance as well. In the two task categories (observable-goal vs unobservable-goal), it chose the non-memory model in the former and the memory model in the latter. The paper observes that Direct “recovers near-oracle success rate while operating at substantially lower average frames processed than the full memory-augmented planner”. It also outperformed simple heuristics (like routing on OOD text alone). In summary, across over 270,000 simulated routing decisions and 245 real-trajectory executions, the direct-router strategy shines. It consistently hugs the ideal success–latency tradeoff curve. Compared to static deployments, Direct matches or exceeds the best success rate with much lower latency (often 30–60% less). The combo of success and efficiency is quantified by a harmonic‐mean metric in the paper, but the intuitive takeaway is simple: treat the planning model as a variable knob, not a fixed choice. Lessons and Outlook. Dao et al. conclude that test-time compute in embodied planning is far from a one-dimensional dial. Chain-of-thought, scaling, and memory each unlock different capabilities, and no single setting is uniformly best. A surprisingly large fraction of tasks need only a cheap model, while the “universal” expensive model is needed only infrequently. Crucially, they show that the scene and instruction themselves often signal the task’s difficulty: even a simple embedding of the input carries enough information for a model to guess whether thinking or memory will help. As the paper puts it, “we can infer task difficulty from scene and instruction context, suggesting [that] practitioners can build effective routers from lightweight, embedding-based features”. In practice, that means one can take a robot’s camera image and command text, run two tiny neural encoders, and have a small classifier decide the compute mode – all in a few milliseconds. This work has immediate practical implications. In any deployed robot system using large VLM planners, energy and time are precious. Instead of always incurring a 20+ second planning overhead, a learned router can yield near-optimal behavior far faster. It effectively brings the “frontier-level” success of billion-parameter models into real-time feasibility. One could imagine extending this idea to more axes (e.g. number of megablock samples, parallelism in beam search, etc.) or even hybrid online/offline strategies. Of course, there are limitations. The router as described must be trained offline on a fixed set of candidate planners. If you want to add a new model or reasoning capability, you need to re-collect performance data and retrain the router. Also, by construction Direct can only choose among existing planners – it does not generate new plans, only selects which algorithm to run. Finally, it treats each planning call independently (it looks at the current scene and instruction only). In very long-horizon tasks, one might imagine richer strategies that condition on the sequence of past decisions; Direct’s router here intentionally avoids that complexity to stay lightweight. In the end, this paper’s key message is that smarter latency-aware planning yields real dividends. Instead of pressing the “max thinking” button for everything, we should let the task decide. Jadee Dao et al. demonstrate a concrete, learned approach to do just that, routing each task’s visual-language query to the cheapest interence configuration capable of solving it. The result is more efficient robots that don’t sacrifice frontier-level capabilities. As they summarize, “treating the high-level planner as a dynamic variable is a practical path to recovering frontier-level success at meaningfully lower cost.”.