Latest / Embodied AI 101 / ACE-Data-0: Turning a Home into a Multimodal Robot-Learning Instrument

Transcript

ACE-Data-0: Turning a Home into a Multimodal Robot-Learning Instrument · Embodied AI 101

• ACE-Data-0: Turning a Home into a Multimodal Robot-Learning Instrument. ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine • Yukang Cao, Haozhe Xie, Beichen Wen, et al. • S-Lab, Nanyang Technological University & ACE Robotics • arXiv preprint • 2026. Release status as of August 3, 2026: The paper and project page are public, and a gated Hugging Face repository is accepting the research license. However, the repository states that the actual data files have not yet been published and that the headline dataset counts will be checked against a final release manifest. So this episode analyzes the documented capture and benchmark, not a release that independent users have already downloaded and reproduced. The Important Part Is Not the 150 Hours. Consider the instruction: prepare a cup of tea and serve it at the table. For a language model, that is a short sequence of tokens. For a household robot, it is a coupled estimation and control problem extending over several minutes. The agent has to remember where the cup is, decide whether to fill the kettle before retrieving the tea, recognize when the water is ready, manipulate multiple articulated or rigid objects, navigate through the room, avoid spilling, clear a destination surface, and determine whether the final state satisfies the instruction. A conventional manipulation dataset might record the grasp of the cup. An egocentric video dataset might record the entire activity but provide no metric state. A motion-capture dataset might recover the body but omit what the person saw. A robot dataset might provide executable joint commands but only for one embodiment and one teleoperation interface. ACE-Data-0 tries to record all these complementary descriptions of an activity at once. Its headline scale is more than 150 hours, 17 million video frames, 200 task categories, 50 participants, two environments, and 75,000 annotated interaction episodes. The more consequential claim, however, is that egocentric video, exocentric video, body motion, articulated hands, object trajectories, audio, and tactile pressure all describe the same physical event on a shared timeline and in a shared coordinate frame. That distinction matters. Embodied learning does not merely need more images of humans manipulating things. It needs correspondences: which visual change accompanied contact, which hand motion moved the object, how the scene state changed, what the actor was trying to achieve, and which earlier action made the current state possible. The best way to understand ACE-Data-0, then, is not as a 150-hour video corpus. It is a high-density measurement corpus. Each hour is intended to contain unusually many aligned targets that can supervise one another. The dataset occupies the opposite end of the spectrum from weakly labeled, internet-scale video: narrow environmental diversity, but exceptionally rich state observability. Two Capture Scales Instead of One Compromised Rig. ACE begins from a sensible physical observation: the sensing geometry needed to resolve fingertip contact is not the sensing geometry needed to track a person crossing an apartment. The project therefore uses two installations. The table-scale site is a roughly 30-square-meter workspace populated with more than 25 interactable instances from over eight object categories. Eight GoPro cameras are placed about 30 to 50 centimeters from the workspace, while 16 OptiTrack PrimeX 22 cameras provide motion capture. This setup concentrates spatial resolution around hands, tools, containers, and small object movements. The room-scale site covers a roughly 200-square-meter furnished apartment containing a kitchen, dining area, living room, and bedroom. Eight ZED One RGB cameras are suspended from a ceiling truss and positioned so that any point in the activity region should remain visible from at least four views. Twelve OptiTrack cameras cover the apartment as one tracking volume. This configuration follows locomotion, posture changes, furniture contact, and object transport across rooms. Participants in both installations wear a head-mounted ACE-Ego-Head-V02 Lite. It contains four fisheye cameras—front-left, front-right, rear-left, and rear-right—recording at 20 frames per second, plus an IMU and five optical markers. The markers let the system measure the headset’s six-degree-of-freedom pose rather than reconstructing it from visual odometry. The participant also wears a 41-marker body suit. Body and object tracking run at 60 hertz. Room-scale hand articulation comes from Manus motion-capture gloves; at table scale, the authors triangulate hand keypoints from the eight exocentric views and manually refine them. Interactable objects are scanned or reconstructed with 2D Gaussian Splatting, attached to optical markers, and tracked as metric six-degree-of-freedom trajectories. Full-palm tactile gloves add pressure grids over both palms and fingers. That last detail about hand ground truth is worth remembering. “Ground truth hand pose” does not have exactly the same provenance across the complete dataset. One portion is glove-derived; another is multi-view triangulation with manual correction. Both are substantially stronger than monocular pseudo-labels, but researchers should retain the acquisition-source metadata and check for systematic differences before treating all hand sequences as identically distributed. The ego and exo streams also play different roles. Ego video gives a robot-like view with large nearby hands and task-relevant attention, but it suffers from head motion, truncation, fisheye distortion, and self-occlusion. Exocentric cameras give stable scene coordinates and whole-body context, but the hands may occupy few pixels or disappear behind the manipulated object. ACE’s paired design makes those trade-offs measurable on identical interactions rather than comparing unrelated datasets. Synchronization Is the Real Engineering Contribution. Multimodal capture is easy to describe and difficult to make scientifically useful. Twelve video streams, motion capture, object trackers, IMUs, tactile arrays, and audio devices all have different clocks, sampling rates, startup delays, and drift. A folder containing these files is not yet a multimodal dataset. ACE uses the 60-hertz OptiTrack clock as its temporal reference. In the room-scale configuration, the eight ZED cameras share a hardware frame trigger and feed a single Jetson Orin host. That aligns the ZED cameras with one another, but not automatically with motion capture. The authors found that the recorded camera timestamps lagged actual exposure by about 290 milliseconds and drifted during a take. Their solution is an optical clock. The motion-capture host displays its own time as machine-readable QR codes. Cameras photograph this display, and the pipeline fits both a constant offset and gradual drift between camera time and OptiTrack time. The important reported number is not the nanosecond resolution of the displayed clock—that should not be confused with nanosecond end-to-end synchronization—but the resulting millisecond-level residual. A separate calibration check estimated camera offsets below four milliseconds. The GoPros in the table-scale setup are first synchronized with one another using their audio tracks. They and the ego cameras observe the same displayed clock. At room scale, each take begins with a deliberate roughly ten-second glance at the clock monitor. Because the four ego cameras share an internal clock and are mutually aligned within two milliseconds, the offset measured from one visible stream transfers to the rest. The tactile gloves require another bridge. At the beginning of a take, the operator performs a short motion while keeping the hand and head approximately rigid. Correlated IMU patterns from the glove and headset establish the tactile-to-ego offset. The final per-take synchronization table maps every ego and exo frame to a corresponding 60-hertz motion-capture frame. Spatial calibration is equally important. The room cameras are widely separated and often share no field of view, which breaks standard calibration procedures based on simultaneous target visibility. ACE instead uses an ArUco board with retroreflective markers at its corners. The RGB cameras observe the printed board, while OptiTrack observes the markers. Motion capture therefore becomes the bridge joining cameras that cannot see one another. After filtering moving-board frames and unreliable edge detections, the system jointly refines board poses and camera parameters. The paper reports median held-out reprojection errors below three pixels, corresponding to roughly a centimeter at typical operating distances. For the moving headset, the system solves a hand-eye calibration between each fisheye camera and the tracked rigid body formed by its five markers. Joint bundle adjustment refines the four camera transformations. The resulting median reprojection error is about two pixels. During a recording, the camera trajectory comes from interpolated OptiTrack measurements rather than image-based SLAM, avoiding cumulative visual drift. This is the most defensible meaning of “data engine” in the paper. ACE is not just an array of sensors; it is a procedure for converting asynchronous measurements into queryable events. At a chosen time, a researcher should be able to retrieve what the person saw, where every tracked body part and object was, whether contact occurred, and how that state projected into each camera. The price is substantial infrastructure. The two sites collectively use 28 OptiTrack cameras and 16 exocentric RGB cameras, along with the wearable systems and recording hosts. One hour produces approximately one terabyte of raw data. This is ambient capture in the sense that the environment observes activity from all directions—not in the sense of lightweight, unobtrusive, commodity collection. From Long Takes to Learning Examples. The behavioral protocol is as important as the sensors. Participants are given goals rather than prescribed motor scripts. They decide where to stand, which route to take, which object to reach for, how to order subtasks, and how to recover from small deviations. ACE divides recordings into three families. Atomic human-object interaction takes contain one to three household tasks and last about three minutes. Examples include pouring, drinking, chopping, cooking, watering plants, and tidying. These are still longer than the isolated grasps common in hand-object datasets, but they are intended to remain relatively self-contained. Chains of interaction last roughly 20 to 30 minutes. They combine multiple activities into one continuous routine and finish with the scene returned to an orderly state. Human-scene interaction takes last about five minutes and emphasize walking, exercising, sitting, leaning, lying down, and making contact with furniture rather than manipulating many portable objects. This design preserves task-level variation that a step-by-step collection script would suppress. Two people can satisfy the same goal through different subtask orders, grasp strategies, paths, pauses, and object choices. Those variations are potentially valuable for learning goal-conditioned behavior because they expose multiple valid trajectories through the same task space. The 75,000 “episodes” should not be interpreted as 75,000 independently collected long-horizon demonstrations. The paper defines an episode as a semantically self-contained subgoal segment inside a longer take. Dividing the approximate 150 hours by 75,000 gives an average of roughly seven seconds per episode. The genuinely unusual asset is that those short segments remain embedded in their preceding and subsequent context rather than being collected as disconnected clips. Each released take is intended to include four ego videos with IMU and headset poses; eight calibrated exocentric videos; 41-joint body skeletons and converted SMPL-X parameters; articulated hand poses; object meshes, poses, boxes, and motion trails; audio; normalized tactile grids; and language at the take and segment levels. Most geometric annotations are computed from physical measurements. Once the cameras, body, and object meshes share a coordinate system, two-dimensional poses, bounding boxes, and trajectories follow by projection. Contact comes from the tactile glove rather than geometric proximity. That matters under occlusion: the label can indicate contact even when neither the contact patch nor the fingertips are visible. Text is the notable exception. The pipeline uses Gemini-3.1-pro-preview to describe ego-video segments, after which human annotators inspect and correct the generated descriptions. Researchers should therefore distinguish three kinds of semantics: the original goal instruction, the derived subgoal structure, and post-hoc visual descriptions. They may all be useful, but they are not interchangeable supervision. The goal expresses intent; the caption describes observed activity. What the Benchmark Actually Reveals. ACE evaluates more than 30 published methods on a held-out ten-hour test set. Its hierarchy moves from low-level tactile inference, through body-state reconstruction, to hand-motion estimation from ego and exo views. This is best understood as an embodied perception benchmark. Despite the language of interaction and robot learning, the reported experiments do not train a policy, transfer a human demonstration to a robot, evaluate task success, or demonstrate improved VLA behavior. The paper establishes whether existing perception systems can recover some of the physical state that a downstream policy would need. Vision Still Does Not Reliably Recover Touch. The tactile track asks a model to predict a full-hand pressure distribution from egocentric video. Temporal accuracy measures whether the model identifies contact at the correct time. Contact IoU evaluates overlap between predicted and measured contact regions. Volumetric IoU also considers pressure magnitude, while center-of-pressure error measures localization. PressureVision produces almost no meaningful spatial overlap. EgoPressureDiff improves contact timing but remains weak on pressure location. TouchAnything performs best, reaching 0.7095 temporal accuracy, 0.1646 contact IoU, and 0.1357 volumetric IoU. Those are clear relative improvements, but the absolute spatial scores remain low. The result captures a recurring distinction in physical perception: recognizing that contact probably happened is much easier than determining which finger region carried load and how pressure was distributed. Visual cues such as halted relative motion, object displacement, and hand configuration can reveal a grasp event. Precise pressure, however, may depend on hidden surface geometry, compliance, friction, and fingertip contact completely occluded by the hand. For robotics, this suggests that vision-to-touch models are currently better viewed as contact priors than substitutes for tactile hardware. They may infer a contact phase or likely region, but the ACE numbers do not support treating predicted pressure as a reliable control signal. Local Body Pose and World Motion Are Different Problems. The human-motion benchmark evaluates 22 methods using single-view exocentric, multi-view exocentric, and egocentric input. It separates local articulation from global trajectory. Procrustes-aligned joint error removes global position, orientation, and scale, asking whether the model recovered the relative body configuration. World-aligned error applies one alignment to the complete path and therefore exposes drift and incorrect placement in the room. SMPLest-X obtains a strong local joint error of 55.7 millimeters after Procrustes alignment. Several contemporary methods remain in the 55-to-65-millimeter range locally. Yet methods that output world trajectories show much larger errors. Human3R, the strongest reported world-frame result, has a world-aligned joint error of 180.2 millimeters. The egocentric EgoAllo and EgoEgo methods reach 252.2 and 306.2 millimeters respectively. The lesson is not simply that pose models are inaccurate. It is that benchmark conventions have historically rewarded a narrower problem. A system can recover a plausible body in every frame while placing that body on the wrong side of a cabinet, drifting through furniture, or accumulating large path errors over several minutes. Scene-aware methods improve world placement more consistently than they improve locally aligned articulation. That makes intuitive sense. Scene geometry constrains where feet, hips, and the body root can exist; it contains less information about the exact bend of an occluded elbow. The benchmark also finds that the available multi-view methods do not automatically beat the strongest single-view systems, which the authors attribute partly to the limited maturity of current multi-view baselines rather than to any inherent weakness of multiple cameras. Egocentric Hand Reconstruction Is Primarily an Egomotion Problem. The hand benchmark provides perhaps the paper’s most interesting controlled comparison. Because ego and exo cameras record the same take, viewpoint changes while behavior and ground truth remain fixed. On ego video, the per-frame WildHands model achieves an 11.2-millimeter Procrustes-aligned joint error, outperforming the video-based Dyn-HaMR and HaWoR on local articulation. But the methods that attempt persistent world-space reconstruction have wrist-trajectory errors of 98.2 and 102.1 millimeters. On exocentric video, WiLoR reaches 9.1 millimeters of aligned joint error. HaPTIC recovers a world-space trajectory with 63 millimeters of error while retaining a competitive 10-millimeter local error. In this evaluation, the fixed, more distant camera beats the moving, close-up camera both in articulation and global trajectory. That result is counterintuitive only if image resolution is treated as the whole problem. Ego video supplies larger hands, but it also couples hand motion with rapid camera motion. A world-space method has to separate how the wrist moved from how the head moved, while handling fisheye distortion, motion blur, and hands leaving the field of view. The exocentric camera starts from a stable coordinate system. ACE also provides the information needed to isolate this failure. The measured headset trajectory can be supplied directly to a hand model, removing visual egomotion estimation from the pipeline. Researchers could then ask whether the remaining error comes from hand reconstruction, camera calibration, or temporal association. Paired exo views and object poses could act as additional privileged signals. The benchmark does not run these fusion experiments, but its capture design makes them unusually clean follow-up projects. Where ACE Sits Among Nearby Datasets. ACE is not the largest dataset in any simple sense. Its contribution is a particular point on the trade-off between behavioral naturalness, metric ground truth, temporal duration, and modality count. At the natural-video end, ACE’s comparison highlights Ego-Exo4D, which contains far more paired first- and third-person activity across many skilled domains and participants. But it does not provide ACE’s combination of measured object trajectories, full-palm tactile pressure, and room-wide metric human motion for each event. Ego-Exo4D is the stronger source of broad activity diversity; ACE is the denser physical instrument. HOT3D is the closest reference point for metric egocentric hand-object tracking. It contains more than 833 minutes and 3.7 million images from 19 participants manipulating 33 rigid objects, with calibrated camera, hand, object, gaze, and scene information. HOT3D offers excellent egocentric geometric supervision, while ACE expands toward exocentric room coverage, whole-body activity, longer routines, audio, and touch. ParaHome is arguably the closest match on contextual household motion. It captures 486 minutes across 207 sequences and 38 participants using 70 synchronized RGB cameras, an inertial body suit, and hand gloves. It includes sequential and concurrent multi-object activity and parameterized articulated objects. ACE is much longer and adds tracked ego views, tactile signals, and audio, while ParaHome’s articulated-object representation covers a class of state changes that ACE explicitly leaves unannotated. DROID illustrates the complementary robot-centered approach. It provides 350 hours and 76,000 robot demonstrations from 564 scenes and 84 tasks, along with the policy-learning code and a reproducible hardware design. DROID actions are directly executable by a corresponding robot; ACE actions are human motions that preserve natural task decomposition but require retargeting. A sensible embodied training stack would use resources like ACE to learn rich human activity and physical-state representations, then connect those representations to robot control through datasets like DROID. How Roboticists Could Use ACE-Data-0. The most immediate use is privileged-sensor learning. During training, a model could receive ego video while exocentric cameras, motion capture, object poses, and touch provide targets. At deployment, only the robot’s onboard camera would remain. This is a natural teacher-student configuration: the instrumented home acts as the teacher, and a sparse onboard sensor suite becomes the student. For hand reconstruction, the measured headset pose can remove camera-motion uncertainty. For human motion, exocentric views and scene geometry can supervise an ego-only body estimator. For contact, tactile grids can label contact onset and release even when the fingers are visually hidden. The same design supports masked-modality objectives in which the model reconstructs missing body, object, audio, or tactile streams from those that remain. A second direction is object-centric human-to-robot translation. Directly cloning human joint trajectories would be a mistake: humans and robots differ in kinematics, reachability, actuation, compliance, and sensing. More transferable targets can be derived from the relationship between hand, object, and scene. Examples include the wrist’s motion relative to an object, approach direction, grasp phase, contact region, object displacement, release condition, and the body motion needed to keep the task reachable. Tactile readings are especially useful for dividing demonstrations into interaction modes: approach, first contact, stable grasp, manipulation, slip or adjustment, and release. These modes can become high-level action tokens or constraints for retargeting. But the glove’s normalized pressure maps should not be confused with complete force-torque measurements. ACE provides rich contact evidence, not full inverse dynamics. The long takes also support world-model research. A model can observe a scene before an action, infer the human’s motion and contact, and predict the resulting object and scene state. The chain tasks add delayed dependencies: an object may leave view and return later, the meaning of an action may depend on an earlier subgoal, and completion requires comparing the final scene with the instructed goal. The key is to preserve the take structure. If the 75,000 episodes are shuffled into independent seven-second clips, much of ACE’s claimed advantage disappears. Training should expose both local interaction windows and longer context containing prior object states, subgoal order, and scene changes. For VLA pretraining, the goal instructions provide task conditioning, while corrected segment descriptions provide denser semantic alignment. Body, hand, and object trajectories can serve as latent-action targets before robot-specific action fine-tuning. A particularly promising design would train an ego-video model to predict an object-centric action representation under privileged exo, motion, and tactile supervision, then connect that representation to robot commands using robot datasets. Finally, ACE offers a better evaluation discipline. Researchers should report not only per-frame aligned pose accuracy, but also world-frame drift, contact timing, contact localization, object-state consistency, and performance as the temporal horizon increases. The paper’s own results demonstrate that a good frame-level metric can coexist with a poor physical trajectory. The Caveats That Matter. The first limitation is diversity. Two highly instrumented sites cannot represent household layouts, lighting, cultures, object distributions, bodies, or routines in general. ACE has high information density per hour but limited environmental coverage. It is better suited to controlled diagnosis and privileged pretraining than to claiming broad household generalization by itself. The visible equipment can also create shortcuts. The head rig, body markers, mocap suit, hand-tracking gloves, tactile gloves, and object markers all appear in the observations. Models may learn dataset-specific visual features, and participants may move differently while instrumented. The paper acknowledges these artifacts, but does not present a domain-transfer experiment showing that models trained on instrumented humans improve perception or control for uninstrumented people or robots. The available scene state is incomplete in ways that matter for household tasks. Tracked rigid objects have excellent geometry and pose, but the dataset does not annotate the changing state of fluids, deformable materials, or articulated mechanisms. A cup can have an accurate six-degree-of-freedom trajectory without a label for how much water it contains. A cloth can move without a tracked deformation state. A cabinet interaction may involve a semantically important joint state that is not represented by a rigid-object pose. This means ACE is not yet a complete world-state dataset. Its strongest state supervision concerns human kinematics, rigid object motion, camera geometry, and contact. Latent properties such as temperature, liquid volume, cleanliness, appliance mode, or deformable configuration may still have to be inferred from video, audio, and language. The benchmark is also narrower than the dataset’s ambitions. It evaluates tactile reconstruction, human motion, and hand motion, but leaves object-pose estimation for future work. It does not benchmark subgoal discovery, task-progress estimation, future state prediction, human-to-robot retargeting, policy learning, or long-horizon robot execution. Claims about imitation learning, world models, and VLAs are plausible use cases, not demonstrated downstream gains. There are measurement questions that future documentation should clarify. The paper reports camera reprojection errors and temporal residuals, but not a complete uncertainty budget across tactile sensing, object-marker occlusion, mocap dropout, hand-glove calibration, or the two different hand-annotation pipelines. A final manifest should ideally expose per-frame validity flags and confidence information rather than presenting all tracked states as equally reliable. And then there is access. As of August 3, 2026, the public repository contains the card and license but says the data shards, loading examples, checksums, storage requirements, and final manifest are still forthcoming. The license is limited to named individuals conducting non-commercial academic research. It prohibits redistribution and commercial use, including commercial product evaluation and internally funded business work; access is personal rather than institutional. Derivative model releases are also restricted to non-commercial academic use under stated conditions. Those restrictions are understandable given that the recordings contain identifiable faces, voices, bodies, and homes. They nevertheless mean that ACE-Data-0 is not an openly reusable foundation dataset in the same operational sense as permissively licensed robot corpora. Industrial teams will need a separate commercial agreement, and academic-industry collaborations should read the definition of non-commercial research carefully. Final Take. ACE-Data-0’s strongest idea is simple: the scarce resource in embodied AI is not merely footage, but synchronized evidence about what a physical interaction looked like, how it moved, what it touched, and what state it produced. Its 150 hours will not compete with web video on visual diversity or with large robot collections on executable actions. What it could provide is a calibrated reference corpus—a physical Rosetta stone connecting first-person observation, third-person context, human motion, object motion, contact, sound, and language. The benchmark already exposes a useful pattern. Present systems often recognize the visible configuration while losing the physical trajectory. They detect that contact occurred without locating pressure. They reconstruct a plausible hand while failing to place it consistently in the world. Those are precisely the gaps that disappear under conventional clip-level, per-frame evaluation and become decisive in a robot. If the released shards, calibration tables, validity metadata, and benchmark tooling match the technical report, ACE-Data-0 could become highly valuable for privileged-supervision experiments and physically grounded evaluation. But the verdict remains conditional: the data were not yet downloadable on August 3, 2026, and the paper has not shown that training on ACE improves an actual robot policy. For now, the work’s most durable contribution is the capture philosophy. Before trying to scale embodied intelligence, build a smaller place where the physical world is observed well enough to reveal exactly what our models still fail to understand.