Latest / Embodied AI 101 / Think Deeply Only When It Matters: DySL-VLA and Action-Aware Dynamic Depth
Transcript
- Think Deeply Only When It Matters: DySL-VLA and Action-Aware Dynamic Depth. DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulation • Zebin Yang, Yijiahao Qi, Tong Xie, Bo Yu, Shaoshan Liu, Meng Li • Peking University and the Shenzhen Institute of Artificial Intelligence and Robotics for Society • arXiv cs.RO preprint • 2026. A robot does not need the same amount of thought for every millimeter of motion. When an arm is sweeping through free space toward a cup, small deviations may be harmless. When its fingers are closing around that cup, the same deviation can decide whether the task succeeds or fails. Yet most vision-language-action models spend essentially the same computation on both moments: the same Transformer depth, the same expensive multimodal backbone, and the same sequence of matrix multiplications. DySL-VLA starts from the proposition that this uniformity is wasteful. Its contribution is not a new general-purpose robot policy or a larger VLA foundation model. It is a conditional-computation retrofit for existing VLAs. The system preserves a small set of important Transformer layers, selectively bypasses the others, and uses recent robot motion to decide when reduced-depth inference is safe. At critical moments, it becomes conservative and restores more of the original network. The public manuscript was first posted on February 26, 2026, with the current arXiv version dated March 17, 2026. Its central result is a 3.75-fold reduction in RoboFlamingo backbone latency on CALVIN while retaining essentially the original task performance. On an OpenVLA-OFT model evaluated with LIBERO, the gain is closer to twofold, including timing on a Jetson Orin. Just as importantly, the conversion trains only routers and adapters rather than extensively fine-tuning the VLA backbone. The deeper lesson is broader than layer skipping. DySL-VLA treats model compute as a control resource that should be scheduled over a trajectory, rather than a fixed cost paid independently at every frame. The Bottleneck DySL-VLA Actually Targets. A conventional VLA receives one or more camera observations and a language instruction, processes them through a vision-language backbone, and emits robot actions such as Cartesian displacement, rotation, and gripper control. In models derived from large VLMs, the language-model portion often dominates both parameter count and latency. The DySL-VLA paper attributes 84.3 percent of OpenVLA inference latency and 75.4 percent of OpenVLA-OFT latency to this backbone. This matters because the natural operating rate of a large VLA is far below the rate expected from a responsive robot controller. The paper contrasts RT-2 at roughly one to three hertz and OpenVLA at roughly three to five hertz with low-level physical interaction commonly requiring tens of updates per second. Action chunking helps by producing several future controls in one forward pass, but it does not make the forward pass itself cheap, nor does it provide fresh visual feedback for every action in the chunk. There are several familiar ways to attack this problem. Quantization makes each operation cheaper. Pruning removes weights or tokens. Distillation trains a smaller policy. Early-exit methods stop before reaching the end of the network. All can work, but they often apply a uniform compute reduction or discard an entire suffix of the model. DySL-VLA instead asks two separate questions:. Which layers can be bypassed without destroying the representation?. At which trajectory steps is bypassing them acceptable?. The first is a model-structure question. The second is a robot-control question. Most of the paper’s novelty comes from keeping those two questions separate and then coupling their answers at runtime. Two Forms of Nonuniformity. The method is built around two empirical observations: Transformer layers are not equally consequential, and robot actions are not equally error-sensitive. Some layers act more like representational landmarks. The authors profile RoboFlamingo models by measuring cosine similarity between layer activations. Their intuition is straightforward: if a layer’s output is extremely similar to its input, that layer is making a comparatively incremental transformation. If the representation changes substantially, the layer may be doing something that later computation depends upon. This is not “information” in a strict information-theoretic sense. Cosine change is a proxy for representational transformation, not mutual information or causal importance. The paper therefore backs up the activation analysis with zero-shot layer-removal experiments. Skipping certain low-similarity, high-change layers causes much larger performance drops than skipping others. Those consequential layers become candidates for the always-executed set. The result is a division between static layers and dynamic layers. Here, static does not mean that the weights are frozen—although the backbone is frozen during DySL training. It means that these layers are statically present in every inference path. Dynamic layers remain available but may be bypassed. The distinction is more useful than simply calling some layers redundant. Directly deleting all the lower-impact layers still destroys performance. Each may contribute only a small refinement, but enough refinements accumulated over a deep network matter. DySL-VLA therefore needs a learned shortcut that approximates the omitted block rather than treating it as an identity map. Some actions have much narrower error margins. The second observation comes from perturbing the VLA during different windows of a manipulation trajectory. In the paper’s cup-and-basket example, perturbations near critical interaction phases degrade task completion more rapidly than perturbations during less constrained motion. Grasping and releasing are obvious examples: an error of a few centimeters in free space may be recoverable, while a similar error at contact can be catastrophic. This is an important difference between dynamic inference for language generation and dynamic inference for control. A weakly predicted text token may lower an evaluation score. A weakly predicted robot action changes the next physical state, potentially invalidating every subsequent observation and action. Errors compound through the environment. The authors then connect action importance to trajectory continuity. Demonstration trajectories tend to be smooth during gross motion because teleoperators move at roughly consistent velocities. Near grasps, releases, alignment corrections, or other fine interactions, trajectories contain stops, direction changes, and micro-corrections. DySL-VLA treats a sudden loss of continuity as evidence that the policy is entering a high-sensitivity phase. That is a heuristic rather than a universal law, but it gives the inference scheduler something valuable: an inexpensive signal derived from controls the system already has. Dynamic-Static Layer Skipping. The architectural core of DySL-VLA is easiest to understand as a sequence of anchor points. Suppose a Transformer has several static layers distributed through its depth. Between each pair of static layers is a run of dynamic layers. The model must always reach the next static anchor, but it has two ways to do so:. Execute the intervening dynamic layers normally. At some dynamic layer, invoke a lightweight adapter and jump directly to the next static layer. The adapter is trained to take the current hidden representation and predict the representation that the omitted dynamic block would have produced. It is therefore a block-level surrogate: the earlier the jump, the more original layers that adapter must summarize. This differs from conventional early exit. An early-exit model stops at an intermediate representation and applies an action head there, discarding all subsequent layers. DySL-VLA can retain important layers near the end of the network even when it skips a large region in the middle. The action head remains attached to the original final representation. It also differs from fine-grained layer skipping in an important systems sense. A router inserted before every layer adds serial overhead. If the router decides to keep the layer, the system pays for both the decision and the original computation. If it skips only one layer at a time, a large fraction of the theoretical savings can disappear into control-flow and adapter costs. By jumping to the next static anchor, DySL-VLA can bypass several layers after a single positive decision. The released RoboFlamingo implementation makes this mechanism concrete. Each eligible dynamic layer receives a small binary router and a residual feedforward adapter. The router mean-pools the token sequence, passes the pooled feature through a small network, and produces skip-versus-keep outputs. Training uses hard Gumbel-softmax decisions, while inference converts the probabilities into binary choices around a 0.5 threshold. If the decision is to skip, the adapter transforms the current activation and execution jumps to the next configured static layer. This produces variable depth within every interval. A simple action might execute a short prefix of dynamic layers and then take the shortcut. A difficult action can traverse the entire interval. Static layers act as representational synchronization points, limiting how far an approximate path can drift before it is processed again by an original, empirically important layer. The paper’s static-layer-ratio ablation suggests that this division is not excessively brittle. On CALVIN, increasing the static ratio from 10 to 30 percent raises latency from 12.6 to 15.8 milliseconds, while average successful sequence length remains between 2.81 and 2.89. The authors choose 20 percent as a reasonable operating point: lower ratios save a little more time but lose some accuracy, while higher ratios add cost with almost no quality gain. That 20 percent should be read as a backbone- and benchmark-specific choice, not a universal constant. Prior-Post Guidance: Scheduling Compute Over a Trajectory. The routers see the model’s current hidden state, but the authors find that routers alone are insufficient. A router trained with pressure to save computation tends to choose shallow paths too often, including at actions where small mistakes cause failure. Running a router before every layer also adds enough overhead to undermine acceleration. DySL-VLA therefore places a temporal control system around the learned routers. The paper calls it prior-post skipping guidance, combining a history-based prior with after-the-fact verification. Pre-skip prediction. At each gap between static layers, DySL-VLA maintains a skipping-allow point. Dynamic layers before this point must execute normally. Their routers are not called. At and after the allow point, routers may decide to jump to the next static anchor. The scheduler updates this point using the recent action trajectory. It considers a window of previous actions and computes how much adjacent action vectors differ. Smooth, similar controls indicate high continuity. Sudden changes indicate lower continuity and potentially greater action importance. When continuity degrades, the allow point moves deeper into the interval. More layers are forced to execute before skipping becomes legal. The size of this forward movement is adaptive: a sharp continuity drop produces a larger shift. When continuity improves, the point retreats gradually, one layer at a time, reopening cheaper paths. This asymmetry gives the mechanism a form of hysteresis. It reacts quickly when the trajectory appears to become difficult, but returns to aggressive skipping more slowly. That helps prevent the scheduler from oscillating between deep and shallow paths around a noisy threshold. It also embodies a sensible control bias: spend extra compute promptly when danger rises, and give it back cautiously after the motion stabilizes. Pre-skip prediction has two jobs. The first is accuracy: it prevents undercomputation during stretches that appear critical. The second is efficiency: when a layer is forced to execute, its router does not need to run merely to say “keep.” This explains the initially counterintuitive ablation in which removing the pre-skip mechanism makes DySL-VLA both less accurate and slower. The continuity window is itself a tuned temporal filter. The paper evaluates windows of one, three, five, seven, and nine actions. Five performs best on CALVIN. A very short window is too noisy and fails to capture a phase transition robustly; a very long one smooths away the change the scheduler is trying to detect. Post-skip verification. History can only indicate that a critical phase is approaching. It cannot always predict the first critical action, because evidence of a trajectory change becomes available only after an action has been generated. DySL-VLA addresses this causal delay with post-skip verification. If a newly predicted action creates the first detected continuity drop, the system reruns that prediction with all layers enabled. It then replaces or verifies the cheap prediction using the full-depth result and recomputes continuity. Conceptually, this is a form of speculative control inference. The model first tries a cheap path. If the output itself indicates that the system may have reached a sensitive phase, it pays for an exact recomputation. Because critical actions are assumed to occupy a relatively small and contiguous portion of a trajectory, occasional re-execution can be cheaper than running the full model everywhere. The ablation supports that trade-off. Removing post-skip verification lowers average latency only from 13.6 to 13.4 milliseconds, but CALVIN sequence length falls from 2.89 to 2.79. In other words, the verifier consumes very little average compute while protecting some disproportionately important decisions. A real deployment does need transactional semantics here. The initially predicted action must be held until verification finishes; once a physical action has been committed, a second forward pass cannot undo it. That requirement is manageable in a synchronous policy loop, but it means the relevant systems metric is not only average latency. Worst-case latency and deadline jitter around critical actions matter too. Training Without Rewriting the Backbone. Freezing the VLA backbone dramatically reduces adaptation cost, but it creates a training problem. Both adapters and routers begin untrained. If they are optimized jointly from the start, the routers quickly discover that skipping through a bad adapter increases task loss. Their safest response is to stop skipping. Once that happens, the adapters receive little useful experience, creating a deadlock. DySL-VLA resolves this with two-stage distillation. In the first stage, it trains adapters without learning routing decisions. For an adapter attached to a given dynamic layer, the frozen original network produces the target activation at the next static anchor. The adapter learns to map its current input directly to that target. This is hidden-state distillation: the shortcut imitates a block of original layers rather than imitating only the final action. Once the adapters can approximate their corresponding blocks, the second stage jointly trains routers and adapters under the task loss. During training, the method selects one candidate skip location between each pair of static layers instead of allowing an early router to monopolize every trajectory. Earlier skip points are sampled more frequently because their adapters must approximate longer blocks and are therefore harder to train. The router’s discrete choice is relaxed into a weighted mixture between the adapter path and the original full-block path. A regularization term favors skipping, with stronger reward for bypassing a longer span. At inference, this soft mixture becomes a hard branch. The resulting adaptation is substantially cheaper than the comparison early-exit system. For the 3-billion-parameter RoboFlamingo experiment, DySL-VLA trains 14 million parameters for 6,700 steps and reports seven GPU-hours. DeeR-VLA fine-tunes 1.2 billion parameters for 92,000 steps and reports 112 GPU-hours. These figures correspond to the paper’s claims of 85.7 times fewer trainable parameters and 13.7 times fewer optimization steps. On the 7-billion-parameter OpenVLA-OFT experiment, DySL-VLA trains 226 million parameters versus 7.1 billion for DeeR-VLA. That is still a meaningful model in absolute terms, but it is far smaller than full-backbone adaptation and leaves the original VLA knowledge largely untouched. What the Experiments Show. The evaluation covers two distinct VLA stacks. CALVIN uses RoboFlamingo 3B and measures the average number of completed tasks in a five-instruction sequence. LIBERO uses OpenVLA-OFT 7B and reports success rates across Spatial, Object, Goal, and Long-Horizon suites. Timing is measured on desktop accelerators and, for OpenVLA-OFT, a Jetson Orin. CALVIN with RoboFlamingo. On the CALVIN D-to-D evaluation, full RoboFlamingo reaches an average successful length of 2.92 at 51 milliseconds of reported backbone latency. DySL-VLA reaches 2.89 at 13.6 milliseconds. On the cross-environment ABC-to-D evaluation, the corresponding figures are 2.85 for RoboFlamingo and 2.83 for DySL-VLA. That is the strongest interpretation of the 3.75-fold result: DySL-VLA nearly reproduces the full model’s task quality while reducing the measured LLM latency from 51 to 13.6 milliseconds. It is not a 3.75-fold speedup with a clear quality improvement over the full model. It is a speedup at approximately equal quality. Against dynamic-inference baselines, DySL-VLA is stronger. DeeR-VLA records 2.83 on D-to-D, 2.82 on ABC-to-D, and 19.3 milliseconds of latency. DySL’s 2.89 versus 2.83 is the reported 2.1 percent relative improvement. FlexiDepth, which lacks DySL’s static anchors and trajectory guidance, reaches only 1.87 and 1.65 at 27.6 milliseconds. Random skipping is fast but collapses to sequence lengths below 0.5, illustrating that speed comes from choosing the path correctly, not merely executing fewer layers. The ablations are particularly informative. Without dynamic-static partitioning, performance falls to the FlexiDepth-like result of 1.87. Without pre-skip prediction, it falls to 2.42 while latency rises to 20.1 milliseconds. Without the two-stage training scheme, task performance remains respectable at 2.82, but latency rises to 74 milliseconds because the routers fail to learn useful skipping behavior. Each component is therefore solving a different failure mode: representational loss, temporal misallocation, and optimization collapse. There is one reporting inconsistency worth flagging. The main CALVIN table lists RoboFlamingo at 2.92, whereas the component-ablation table lists it at 2.82. This may reflect different runs or evaluation settings, but the manuscript does not explain it in the surrounding text. The speed comparison remains numerically clear because both tables use 51 milliseconds, but tighter reporting would make the accuracy comparison easier to audit. LIBERO with OpenVLA-OFT. The OpenVLA-OFT result is less dramatic but arguably more deployment-relevant. Full OpenVLA-OFT averages 97.1 percent success and takes 53 milliseconds on an A6000 or 676 milliseconds on Jetson Orin. DySL-VLA averages 96.5 percent, with latency of 27.4 milliseconds and 345 milliseconds, respectively. So the trade is roughly a twofold reduction in backbone latency for a 0.6-percentage-point drop in average success. DySL-VLA also exceeds DeeR-VLA’s 95.3 percent average while running faster, and it avoids the severe degradation suffered by the direct FlexiDepth adaptation, particularly on LIBERO-Goal and LIBERO-Long. The paper translates 345 milliseconds per OpenVLA-OFT inference into a 23.2-hertz control rate because each inference predicts a chunk of eight actions. That arithmetic is correct as an action-output rate: about 2.9 chunks per second multiplied by eight actions per chunk. It should not, however, be confused with 23.2 full VLA replans from fresh visual observations every second. Action chunking raises setpoint throughput by amortizing expensive inference over several future controls. The reported timings are also framed as LLM-backbone latency rather than complete sensor-to-actuator loop time. Vision encoding, action decoding, host-device transfer, camera acquisition, safety checks, and robot communication still remain. The experiments establish that the targeted bottleneck becomes substantially cheaper; they do not by themselves establish the same multiplicative gain for an entire deployed control stack. Where DySL-VLA Fits Among Conditional-Compute Methods. RoboFlamingo provides one of the paper’s starting points: it adapts OpenFlamingo for manipulation, combining pretrained visual-language understanding with a policy head that models temporal history. DySL-VLA leaves that policy structure largely intact and accelerates its expensive language backbone. OpenVLA-OFT attacks a different axis. Its optimized fine-tuning recipe uses continuous actions, parallel decoding, and action chunking, producing several actions in one pass. DySL-VLA then reduces the depth cost of that pass. The two methods are complementary: OFT asks how many controls can be produced per inference, while DySL asks how much Transformer depth each inference needs. FlexiDepth is the closest architectural ancestor. It equips pretrained LLMs with routers and adapters so that different generated tokens can use different amounts of depth. DySL-VLA adds the static-anchor structure and replaces token difficulty as the sole routing signal with a trajectory-level notion of action importance. Mixture-of-Depths operates along yet another axis, selecting which token positions receive computation at each layer under a predefined budget. Its routing keeps total tensor shapes predictable while changing the identities of processed tokens. DySL-VLA instead changes how many layer blocks are traversed at a robot time step, making it more naturally suited to batch-one edge inference but potentially less friendly to large batched serving. DeeR-VLA represents the early-exit alternative. It can allocate more computation to difficult actions, but exiting removes all remaining layers, including potentially informative late layers. Recovering performance requires intermediate action heads and extensive backbone training. DySL’s anchors allow it to preserve important late transformations while using much lighter adaptation. Quantization, token pruning, and dynamic depth should not be viewed as mutually exclusive. One changes operation precision, another changes sequence width, and the third changes executed depth. In principle, a deployed VLA could use all three, although gains will not necessarily multiply because each optimization changes the relative cost and overhead of the others. The Important Limitations. The first limitation is the definition of action importance. Trajectory discontinuity is clever and nearly free, but it is a proxy shaped by teleoperation data. Some safety-critical operations are smooth: insertion, cutting, surface following, or moving a fragile object at constant velocity. Conversely, collision avoidance or recovery can produce jerky but nonterminal motion. The metric also combines heterogeneous action dimensions. Translational displacement, rotation, and binary gripper changes may have different scales and consequences. Without careful normalization or weighting, one channel can dominate continuity. A more general scheduler might combine motion history with predicted contact, force-torque signals, visual uncertainty, action-distribution entropy, ensemble disagreement, constraint margins, or a learned estimate of failure sensitivity. Second, post-skip verification moves extra computation into exactly the moments with the tightest deadlines. Average latency can remain excellent while tail latency increases. A manipulation system operating under hard real-time constraints needs distributions of latency, not only means: percentiles, deadline misses, recomputation frequency, and the duration of full-depth bursts. Third, layer skipping reduces arithmetic but not backbone storage. The full model must remain resident because any action may require all its layers. The released OpenVLA-OFT setup still specifies roughly 16 GB of GPU memory for LIBERO inference. DySL-VLA is therefore an inference-compute method, not a solution to model-memory constraints. Dynamic branches can also complicate batching and compilation. Two robot observations in the same server batch may choose different layer paths, reducing GPU utilization. This matters less for one robot running batch-one inference on an edge device—the scenario emphasized here—but more for centralized robot fleets or large-scale simulation. Fourth, the task evaluation is simulated. The paper times the system on Jetson Orin, but the reported manipulation results come from CALVIN and LIBERO rather than a physical robot campaign. Simulation is appropriate for controlled comparison, yet real contact dynamics, camera delays, action noise, and recovery behavior are precisely where the continuity heuristic needs the strongest validation. Finally, the static-layer profiling procedure may need to be repeated for every backbone and adaptation. Activation change is architecture-, data-, and potentially task-dependent. The paper demonstrates the approach across RoboFlamingo and OpenVLA-OFT, which is valuable, but it does not establish that one static-layer map transfers across robot embodiments, fine-tuned checkpoints, or newer diffusion and flow-based action models. How a Robotics Team Might Use It. For a team adapting this idea, the right workflow begins with profiling rather than immediately training routers. First, measure adjacent-layer activation change on representative robot data, then validate the ranking causally by bypassing individual layers. Choose static anchors from layers that both transform the representation substantially and cause clear degradation when removed. The paper’s 20 percent ratio is a useful initial budget, but latency and quality should determine the final choice. Second, train each shortcut adapter against the frozen backbone before introducing routing. This part of DySL-VLA is broadly applicable: whenever a learned router depends on initially unreliable experts, warm up the experts first. Third, design the importance signal for the embodiment. Action continuity may be enough for free-space pick-and-place. Contact-rich tasks should probably include force, tactile, or predicted-contact signals. Safety-critical applications may define regions in which skipping is forbidden regardless of router confidence. Fourth, make verification transactional. Generate the candidate action, evaluate the verification trigger, and only then commit the action to the controller. If action chunks are used, decide whether verification protects the entire chunk, only its first action, or a receding-horizon prefix. Fifth, benchmark the whole control loop. Report end-to-end latency, action and observation rates separately, recomputation frequency, energy, memory, and tail latency. Quality evaluation should include perturbations that break the heuristic: smooth precision tasks, discontinuous recovery motions, unseen embodiments, and noisy low-level controllers. The official repository provides separate RoboFlamingo and OpenVLA-OFT implementations, along with LIBERO checkpoints. The OpenVLA path currently requires replacing the installed Transformers Llama modeling file, so production users will probably want to isolate the changes behind a custom model class rather than patching a shared Python environment. The Bottom Line. DySL-VLA is compelling because it treats VLA acceleration as more than generic model compression. Its compute policy is aware of both where information is transformed inside the network and when precision matters along the robot trajectory. Static layers preserve representational landmarks. Dynamic layers provide elastic depth. Adapters make long jumps possible. Motion history tells the model when to become conservative, and post-skip verification catches the first critical action that history could not predict. Two-stage distillation makes the whole mechanism trainable without rewriting the underlying VLA. The reported 3.75-fold result should be understood precisely: it is a large reduction in RoboFlamingo backbone latency at nearly equal CALVIN performance, not a universal end-to-end robot speedup. The OpenVLA-OFT results suggest that less redundant, already optimized models still have meaningful depth to remove, though closer to a factor of two. Most importantly, DySL-VLA points toward a useful design principle for embodied AI: compute should follow the structure of physical interaction. A robot should think cheaply while the world is forgiving—and spend its full model only when the next action truly matters.