Latest / Embodied AI 101 / LDA-1B and the Case for Never Throwing Robot Data Away

Transcript

LDA-1B and the Case for Never Throwing Robot Data Away · Embodied AI 101

• LDA-1B and the Case for Never Throwing Robot Data Away. LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion • Jiangran Lyu et al. • Peking University, Galbot, CASIA, BAAI, Tsinghua University, Sun Yat-sen University, and NVIDIA • Robotics: Science and Systems, RSS 2026 • 2026. Publication note: The primary record lists the first arXiv submission on February 12, 2026, a revised version on June 3, 2026, and acceptance at RSS 2026. It does not list August 12, 2026 as the paper’s original publication date. The Paper’s Real Thesis. Here is the one-sentence pitch for LDA-1B: instead of asking whether a trajectory is good enough to imitate, ask what that trajectory is good enough to teach. That sounds like a small reframing, but it attacks one of the most expensive assumptions in contemporary robot foundation models. Behavior cloning treats action labels as the primary source of supervision. If a demonstration contains pauses, retries, awkward motions, or outright failures, training on it can teach the policy to reproduce those behaviors. The usual response is therefore to filter aggressively, retain the expert subset, and discard the rest. LDA-1B argues that this throws away a great deal of embodied information. A clumsy attempt to close a drawer might be a poor policy target, but it still records how the drawer responds to contact. A failed grasp still shows the relationship among hand pose, object motion, and slippage. A first-person human video without robot actions cannot directly supervise a robot controller, but it can still teach what state transitions are plausible while someone manipulates an object. The authors turn that observation into a role-aware training regime. High-quality trajectories supervise the policy as well as dynamics. Lower-quality trajectories are routed toward objectives that do not require optimal behavior. Actionless videos supervise visual forecasting. Everything is learned by one multimodal diffusion transformer operating over action and visual-feature streams. The result is a nominally one-billion-parameter dynamics-action model pretrained on more than 30,000 hours of heterogeneous robot, simulation, and human experience. The most important idea in LDA-1B is therefore not the parameter count. It is the proposal that data quality should determine the objective applied to a sample, rather than whether that sample enters training at all. Why Imitation-Only Scaling Wastes Embodied Experience. A conventional vision-language-action policy learns a mapping from observations and instructions to actions. This is attractive because the deployment interface is simple: show the model the scene, tell it the task, and execute the predicted action chunk. But the supervision is semantically demanding. The action target is implicitly supposed to represent what the robot should do, not merely what it did. That distinction matters when datasets become large and heterogeneous. Teleoperation logs contain hesitation and recovery. Automatically generated trajectories may be successful but mechanically unnatural. Human motion can express the right intent using kinematics that a robot cannot directly reproduce. Videos may contain no machine-readable actions whatsoever. If all of these examples are forced through the same action-imitation loss, more data can make the policy worse rather than better. Dynamics learning has a weaker requirement. To learn a forward model, the action does not need to be optimal; it only needs to be paired with the transition it produced. To learn inverse dynamics, the model can study which motion connected two observed states without treating that motion as the ideal response to an instruction. Visual forecasting needs neither robot actions nor expert execution. It asks what future state is plausible given the scene and task context. This is the conceptual opening inherited from Unified World Models, or UWM. UWM couples action and video diffusion so that one model can act as a policy, forward model, inverse model, or video generator by changing which modalities are supplied and which are denoised. It also makes action-free video useful to policy pretraining. LDA keeps that four-way formulation but changes the representation, architecture, dataset engineering, and treatment of data quality in an attempt to scale it to foundation-model territory. A useful way to think about the difference is that behavior cloning assigns one meaning to an action sequence: “imitate this.” LDA can assign at least two meanings: “imitate this if it is good,” or “understand the physical consequences of this whether or not it is good.” That separation is especially valuable during real-world collection, where the marginal cost of recording an imperfect attempt is often low but the cost of producing another pristine expert demonstration is high. One Denoiser, Four Embodied Questions. LDA trains a single model to answer four related questions:. Policy: Given the current visual context and instruction, what action chunk should the robot execute?. Forward dynamics: Given the current context and a future action chunk, what future visual state should result?. Inverse dynamics: Given the observed transition, what action likely caused it?. Visual forecasting: Given the current scene and instruction, what future visual state is likely, even when no action annotation exists?. The implementation is a flow-matching diffusion model. During training, target actions and future visual features are mixed with noise. The network receives a sampled noise level and learns the direction that moves the noisy variables toward the clean target. At inference, it starts from noise and repeatedly applies the learned vector field to produce an action chunk or future visual representation. Four learned task embeddings tell the transformer which question it is answering. Two register tokens act as placeholders when a modality is absent. During policy training, for example, the model denoises actions while a visual register stands in for the unobserved future visual target. During actionless visual forecasting, it denoises future visual features while an action register occupies the missing action stream. This preserves one network topology rather than requiring a separate architecture for every objective. The routing policy is the crucial part. High-quality human and robot demonstrations can activate all applicable objectives. Lower-quality trajectories are kept away from direct instruction-conditioned imitation and used primarily for dynamics and forecasting. Actionless human video activates visual forecasting. Losses are switched on selectively, so the model receives only the supervision a sample can plausibly support. This is more principled than simply assigning low weights to noisy demonstrations. A reduced behavior-cloning weight still says that the action is a somewhat desirable target. LDA instead changes the semantic task. The action can be treated as an intervention whose effects should be modeled, rather than as behavior the deployed policy should reproduce. There is also an important boundary around the term “world model.” LDA learns predictive latent dynamics, but the reported controller does not perform model-predictive control by searching over candidate actions and rolling out their consequences. At deployment, the policy objective directly denoises an action chunk. The dynamics tasks shape the shared representation during training; they are not shown operating as an explicit online planner. The Central Representation Bet: Predict DINO Futures, Not Pixels. The largest technical departure from earlier unified video-action models is the future-state representation. Pixel prediction asks the model to reproduce everything visible: lighting, texture, shadows, camera noise, background clutter, and small viewpoint changes. Even a VAE-compressed video target retains much of this appearance burden. For robotics, however, many visually expensive details are causally irrelevant. The controller usually cares more about object identity, geometry, contact regions, and motion than about reconstructing the exact wood grain on a table. LDA therefore predicts frozen DINOv3 visual features. The paper’s argument is that these features retain spatial and object-level structure while suppressing nuisance variation. Forward dynamics becomes the problem of predicting how a structured feature map changes under an action, rather than rendering a photorealistic future frame. The output is still spatial: the reported latent shape is 14 by 14 by 384, corresponding to 196 feature locations rather than one global embedding. This representation choice dominates the simulation ablations. A one-billion-parameter UWM using VAE-derived visual latents reaches roughly 19% average success on RoboCasa-GR1, and replacing its standard transformer with LDA’s multimodal transformer moves it only to 20%. LDA with DINO features reaches 55.4%. By comparison, growing LDA from 0.5 billion to one billion parameters yields a smaller increase, from 50.7% to 55.4%. The authors interpret this as evidence that a larger model helps only if the prediction space exposes the right structure. There is a broader lesson here. Robot world models need not be miniature video generators. If the intended use is representation learning and control rather than human-viewable simulation, an abstract predictive space may be more compute-efficient and more transferable. The trade-off is that the model becomes dependent on whatever information the frozen encoder preserves. Dynamics that are visually subtle, such as force, friction, incipient slip, or compliance, may not be represented reliably just because they matter to the controller. LDA also treats vision and action as asynchronous streams. Raw data is standardized at 10 hertz, actions are modeled at 10 hertz, and visual features are sampled at 3 hertz. The model predicts 16-step action chunks, or about 1.6 seconds of control at the nominal action rate. This gives the action stream sufficient temporal resolution without spending transformer capacity on nearly identical consecutive image features. A short two-timestep history of observations and actions provides temporal context. A 1B Dynamics-Action Core, Not a 1B Total System. The model name needs one qualification. LDA-1B contains a roughly one-billion-parameter trainable MM-DiT and action stack, but the full system also uses Qwen3-VL-4B-Instruct for semantic conditioning and DINOv3-ViT-S for visual features. The paper’s model-comparison table explicitly counts trainable parameters while excluding frozen components. During pretraining, both Qwen3-VL and DINOv3 are frozen; during downstream fine-tuning, the VLM is unfrozen. Calling this a one-billion-parameter dynamics-action core is therefore more precise than calling the entire deployed system a one-billion-parameter network. The MM-DiT has 16 layers, a hidden width of 1,536, and 32 attention heads. Action and visual tokens use modality-specific projections and feed-forward networks, preserving separate inductive biases. They interact through shared attention, while VLM-derived tokens provide semantic and instruction context. Diffusion timestep and task identity modulate each layer through adaptive normalization. Separate output heads decode the denoised actions and predicted DINO features. This separation is sensible for asynchronous modalities. A visual patch and a robot joint command should not necessarily pass through identical projections merely because both have been converted into tokens. At the same time, contact reasoning requires cross-modal interaction: the meaning of an action depends on where the hand is relative to the object, and the relevant visual region depends on the candidate action. Shared attention gives the streams a place to negotiate that relationship. The action representation is likewise designed for cross-embodiment transfer. Actions are expressed around the hand or end effector using wrist motion plus finger configuration. A parallel gripper contributes its width; a dexterous hand is represented through finger keypoints in the wrist frame. Human trajectories use six-degree-of-freedom wrist poses and MANO hand parameters. The intention is not to pretend that a human hand and every robot hand are kinematically identical, but to align them around a common geometric description of manipulation. Pretraining is reported to use 48 NVIDIA H800 GPUs for 400,000 iterations, at a total cost of 4,608 GPU-hours. The global batch configuration is listed as 32 times 48. The optimizer is AdamW with a learning rate of (10^{-4}), cosine decay, and a minimum rate of (5\times10^{-7}). The paper does not provide enough throughput and data-mixture detail to independently reconstruct the complete compute accounting, but it does indicate that the large pretrained encoders remain frozen so the expensive optimization is concentrated in the dynamics-action stack. EI-30k: Standardization May Be the Bigger Contribution. A universal objective is not useful without a universal data interface. EI-30k is the authors’ attempt to build one. The collection combines four broad categories: real robot trajectories, simulated robot trajectories, egocentric human data with estimated or recorded actions, and actionless egocentric human video. The main paper reports approximately 8,000 hours of real robot data, 8,600 hours of simulation, 7,200 hours of action-labeled human data, and 10,000 hours of actionless video. Representative sources include Open X-Embodiment, Agibot World, InternData-A1, BEHAVIOR-1K, Ego4D, Ego-Exo4D, EgoDex, and the authors’ Egocentric-10k collection. All sources are converted into LeRobot 2.1 format. The standardized records include end-effector or wrist pose, finger articulation or gripper state, camera calibration, timestamps, task identity, and episode boundaries. Sequences are initially resampled to 10 hertz. Dataset-specific configurations define modality keys, temporal offsets, normalization, camera handling, and robot embodiment tags. The most labor-intensive step appears to be coordinate alignment. Robot and human datasets often disagree about where the end-effector frame lies, which axes point where, and whether poses are expressed in camera, base, or world coordinates. LDA introduces dataset-specific rigid offsets to place wrists and grippers into a common canonical frame. For moving egocentric cameras, hand trajectories are reprojected to decouple camera motion from hand motion. Human hands are converted to a common 21-keypoint representation when possible, while gripper widths and robot joint encodings are normalized across platforms. The data is also cleaned semantically. Idle and head-only segments are removed. Hand visibility and kinematic validity are checked. Language descriptions are normalized, and a VLM is used when annotations are inconsistent or missing. Every trajectory receives a quality label based on factors such as action accuracy and annotation completeness. Crucially, low-quality examples are retained rather than deleted. For practitioners, this pipeline may be more consequential than the exact transformer block. Cross-embodiment learning fails quickly when action coordinates have inconsistent signs, origins, units, or semantics. A sufficiently large network cannot reliably discover that one dataset defines positive rotation in the opposite direction or that another measures gripper state as commanded effort rather than aperture. LDA’s results reinforce the unglamorous point that foundation-model scaling depends on careful schema engineering. There are nevertheless unresolved data questions. The category totals in the main text and the detailed appendix itemization do not line up perfectly, although both support an aggregate of roughly 34,000 hours. More importantly, hours of actionless video, simulation, noisy robot exploration, and expert teleoperation are not interchangeable units of information. The paper does not report an effective-token accounting or enough detail about the final pretraining mixture to determine how heavily each source influenced optimization. As of August 16, 2026, the project page still marks the EI-30k data as “coming soon,” and the repository lists pretraining data and preprocessing scripts among unfinished release items. That means the dataset quality labels, alignment decisions, and full ingestion pipeline—the ingredients most central to the paper’s claim—cannot yet be independently audited or reproduced. What the Experiments Actually Establish. Simulation: Dynamics Helps Most When Actions Have Consequences. The primary simulation evaluation uses RoboCasa-GR1: 24 kitchen manipulation tasks executed by a humanoid with dexterous hands from egocentric RGB observations. Each model is fine-tuned on 1,000 demonstrations per task and evaluated over 51 trials per task. LDA-1B averages 55.4% success, compared with 47.6% for GR00T-N1.6, 47.8% for StarVLA, and 51.3% for the authors’ stronger GR00T-EI10k reproduction pretrained on EI-30k’s high-quality subset. The GR00T-EI10k comparison is especially informative. Simply replacing the baseline’s original pretraining with a cleaner subset of the authors’ data raises performance from 47.6% to 51.3%. LDA then adds another 4.1 percentage points under the same nominal one-billion-parameter trainable budget. In other words, part of the headline gain comes from data curation, but the unified dynamics objectives still contribute beyond the stronger data baseline. The largest task-level improvements occur when placement must be followed by a mechanically consequential closing or retraction motion. LDA scores 76% versus GR00T’s 51.5% on placing a bottle in a cabinet and closing it, 71% versus 13% on placing a can in a drawer and closing it, and 52% versus 14% on placing milk in a microwave and closing it. The authors show failure cases where GR00T places an object successfully and then collides with it during arm withdrawal, while LDA produces a trajectory that preserves the achieved state. LDA is not universally better. GR00T remains ahead on some simpler transfer tasks, such as moving an object from one plate to another. That is consistent with the proposed mechanism: explicit dynamics modeling should matter most when contact, articulated objects, clutter, or post-placement interactions make the consequences of an action difficult to ignore. Real Robots: The Gains Grow With Contact and Dexterity. The physical experiments use Galbot G1 and Unitree G1 humanoids with parallel grippers, 10-degree-of-freedom BrainCo hands, or 22-degree-of-freedom SharpaWave hands. Policies receive only egocentric RGB from a head-mounted camera. Galbot is excluded from EI-30k pretraining, so its experiments also test adaptation to a new embodiment. The authors collect 100 teleoperated trajectories per task without demanding expert execution; around 50% to 80% are categorized as high quality. On the gripper tasks, LDA reaches 90% on handover, compared with 70% for (\pi_{0.5}) and 50% for GR00T. It reaches 60% on flipping a box, versus 20% for GR00T, and 80% on the pouring task, 20 points above the strongest baseline. On the long-horizon rubbish task—which combines bimanual manipulation, a dustpan, sequential collection, and disposal—LDA receives a 35% score while both baselines receive zero. The dexterous-hand results are more dramatic. With the lower-DoF hand, LDA scores 90% on bottle placement and 80% on extracting a nail with a claw hammer; GR00T reaches 75% and 40%, while (\pi_{0.5}) reaches 20% and zero. With the 22-DoF hand, LDA scores 70% on picking bread and 90% on flipping bread with a spatula. Both baselines reach only 10% on the flipping task. These experiments support the paper’s central narrative: dynamics-oriented pretraining appears increasingly useful as action dimensionality, continuous contact, and error accumulation become more important. But the comparison evaluates the complete LDA recipe, not architecture alone. Baselines are fine-tuned only on the filtered expert subset, while LDA consumes all 100 trajectories through its role-aware objectives. That is appropriate for testing whether the method can exploit noisy data, but it means the real-world headline combines a better learner with access to more fine-tuning samples. The Cleanest Test of Mixed-Quality Ingestion. The paper isolates the noisy-data question on two tasks. For placing a pen into a box, LDA improves from 70% using 63 high-quality trajectories to 80% after adding 37 low-quality trajectories. On bimanually removing a lid, it improves from 50% to 60% after adding 34 low-quality examples. Under the same comparison, (\pi_{0.5}) falls from 60% to 40% on the pen task and from 50% to 40% on the lid task. This is probably the most direct evidence for universal ingestion, but it is also narrow: two tasks, roughly 100 examples each, and small real-world evaluation sets. The next step should be a larger study varying the fraction, type, and severity of low-quality behavior. Pauses, recoverable mistakes, incorrect task completion, calibration errors, and random exploration are all “low quality” in different senses, and they may not be equally useful for dynamics. Is This Really Scaling?. LDA’s scaling study varies both data volume and model size. Evaluation uses open-loop action L1 error on a held-out Agibot World subset. Policy-only training improves at first but becomes unstable when lower-quality data is added. Partial co-training with forecasting or dynamics is more robust, while the complete four-objective recipe improves consistently from roughly 5,000 to 30,000 hours. Even the final addition of 10,000 hours of actionless human video reduces action error, ultimately reaching 6.6. Increasing the trainable model from 0.1 billion to 0.5 billion and then one billion parameters also monotonically reduces prediction error under full co-training. That is encouraging evidence that the architecture can use additional capacity rather than merely tolerate it. Still, this is a scaling trend rather than a robotics scaling law. The paper does not fit power laws, estimate compute-optimal trade-offs, or show closed-loop success curves at every data and model scale. Action L1 error is reproducible and convenient, but two policies with similar open-loop error can behave very differently after their mistakes alter the observation distribution. The strongest closed-loop evidence remains the endpoint comparison among the final systems. Where LDA Sits Among UWM, UVA, π0.5, GR00T, and Motus. LDA’s closest conceptual ancestor is UWM. Both models use joint action-and-visual diffusion to support policy learning, forward dynamics, inverse dynamics, and future observation generation. LDA’s contribution is to make that formulation more scalable by replacing pixel-oriented latents with DINO features, separating modality experts within MM-DiT, explicitly routing data by quality, and assembling a much larger standardized corpus. The Unified Video Action model, or UVA, pursues a related goal through a joint video-action latent and separate lightweight decoding heads. Its decoupled design lets action inference bypass video decoding, while masked training enables policy, forward dynamics, inverse dynamics, and video prediction. LDA similarly avoids generating viewable video during policy inference, but places more emphasis on heterogeneous data ingestion and a pretrained semantic visual target. (\pi_{0.5}) also demonstrates that heterogeneous co-training can improve open-world robot control. It mixes robot data with high-level subtask prediction, object localization, language supervision, and web tasks. The difference is one of emphasis. (\pi_{0.5}) broadens the policy’s semantic and task-level knowledge; LDA tries to extract physical transition knowledge from examples that should not directly supervise optimal action selection. These are complementary directions rather than mutually exclusive ones. GR00T provides the main foundation-policy reference point for humanoid control. LDA’s comparison suggests that adding explicit predictive dynamics is particularly valuable for contact-rich and dexterous tasks, though its reproduced GR00T-EI10k baseline also shows how much performance can come from improving the pretraining corpus alone. Motus is another closely related unified world-action model. It combines understanding, video generation, and action experts and uses latent actions to access heterogeneous motion data. Relative to that line, LDA’s identity is unusually concrete: structured DINO prediction, manually aligned hand-centric actions, quality-conditioned objective routing, and a large LeRobot-formatted data mixture. How You Might Adapt LDA. The public repository provides training and evaluation code for RoboCasa-GR1, an open-loop evaluation interface, and instructions for adding real-robot datasets. The software expects Python 3.10, FlashAttention 2, Qwen3-VL, DINOv3, and LeRobot-formatted data. The default scripts expose the four task weights, observation history, future-observation offset, action dimensionality, embodiment registry, and data-mixture configuration. For a new robot, the most important adaptation step is likely not choosing a learning rate. It is defining action semantics. You need a canonical end-effector frame, a decision about absolute versus delta actions, a representation for fingers or grippers, consistent units and rotation conventions, and an auditable transform from the native controller state into that space. Only after that should you register the embodiment, specify modality and temporal keys, choose normalization, and add the dataset to a mixture. The second step is to classify supervision rather than just trajectories. A single episode may contain segments suitable for imitation, segments useful only for dynamics, and idle intervals that should be removed. LDA uses trajectory-level quality labels, but a production implementation could benefit from segment-level routing or soft objective weights. Recovery behavior is a particularly interesting category: blindly imitating the failure that preceded recovery is undesirable, but learning the recovery itself may be extremely valuable. The released recipe is most immediately usable as a fine-tuning framework. Reproducing EI-30k pretraining is not yet turnkey because the unified data, preprocessing scripts, and full mixture are unavailable. Although checkpoint links are present, the pretraining model card is currently empty, and the repository’s release checklist remains inconsistent about checkpoint availability. Teams evaluating adoption should verify artifact versions and licenses rather than assuming the project page, repository, and model host are fully synchronized. What Remains Uncertain. The first open question is whether DINO features contain enough state for control outside visually dominant manipulation. The authors themselves identify the fixed representation and predominantly egocentric viewpoint as limitations. Multi-view geometry, depth, proprioception, force, tactile sensing, and audio would require new modality experts or predictive targets rather than simply more RGB pretraining. Second, latent prediction is not yet demonstrated as planning. LDA shows that auxiliary dynamics objectives can improve a direct policy, but it does not establish that the model can perform reliable long-horizon counterfactual rollouts, score candidate plans, or compose imagined transitions beyond its training horizon. “Dynamics-aware policy” is fully justified; “general-purpose planner” would not be. Third, the real-world sample sizes are modest. Many tasks use 10 or 20 trials, some use partial-credit metrics, and no confidence intervals are reported. The improvements are often large enough to remain interesting, but small differences should not be overinterpreted. Evaluation against baselines also mixes model effects with LDA’s ability to consume additional low-quality fine-tuning data. Fourth, the scaling argument relies on open-loop action error. Future work should report closed-loop curves as data and model size increase, alongside calibration of the latent forward model, rollout consistency, failure-recovery behavior, and inference latency. Diffusion policies can be practical with few denoising steps, but the paper does not provide a full control-loop throughput analysis. Finally, universal ingestion depends on labels that say which data is safe for which objective. The paper describes quality annotation at a high level, but the exact thresholds, inter-annotator consistency, automated scoring, and sensitivity to routing mistakes remain unclear. If those decisions require extensive manual judgment, part of the apparent reduction in data curation cost may simply have moved from filtering to annotation. The Takeaway. LDA-1B is compelling because it proposes a better unit of value for embodied data. A trajectory does not have to be worth imitating to be worth keeping. Robot mistakes can teach consequences, human motion can teach manipulation structure, and actionless video can teach plausible futures—provided the learning system routes each example to an objective it can honestly supervise. The broader blueprint is straightforward: align actions geometrically, predict futures in a task-relevant latent space, separate policy optimality from transition validity, and scale model capacity only after the data interface is coherent. If EI-30k and its preprocessing pipeline become fully available, LDA may prove to be as much a data-systems contribution as a model contribution. Until then, the strongest result is also the central challenge: the paper shows that universal embodied ingestion can work, but the field still needs the released infrastructure to determine how universally it can be reproduced.