Latest / Embodied AI 101 / Catching on Optical Expansion: Inside Pixel2Catch's Monocular Sim-to-Real Pipeline

Transcript

Catching on Optical Expansion: Inside Pixel2Catch's Monocular Sim-to-Real Pipeline · Embodied AI 101

• Catching on Optical Expansion: Inside Pixel2Catch’s Monocular Sim-to-Real Pipeline. Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera • Seongyong Kim, Junhyeon Cho, Kang-Won Lee, and Soo-Chul Lim • Dongguk University • IEEE Robotics and Automation Letters, 11(8), 9287–9294 • 2026. Why catching exposes the entire robotics stack. Catching a thrown object is an unusually compact test of embodied intelligence. The robot has only a short observation window, perception errors accumulate directly into interception errors, and the controller cannot simply correct after contact. It must place the end effector in the right region, orient a high-degree-of-freedom hand, time closure, absorb impact, and retain the object. A system can solve reaching but fail at grasping, or detect the object accurately but react too late. Most robotic catching pipelines therefore reconstruct some explicit geometric state: a 3D object position, a ballistic trajectory, an impact point, or all three. That state is normally obtained from motion capture, stereo or RGB-D sensing, event cameras, or multi-view tracking. The control problem then becomes one of intercepting a predicted trajectory. Pixel2Catch asks whether that explicit reconstruction step is always necessary. Its alternative is deliberately modest: segment the object in the image, reduce the segment to a bounding box, measure how the box center and scale change, and let a reinforcement-learning policy discover how those projective cues map to arm and finger motions. The second idea is to divide the physical system into two learning agents—one for the arm and one for the hand—while training them cooperatively in a single stage. The resulting policies are trained entirely in Isaac Lab and deployed on a real UR5e–Allegro system without real-world policy fine-tuning. The most important interpretation is that Pixel2Catch is not an end-to-end visual policy. It is closer to learned image-based visual servoing, with a carefully chosen low-dimensional perception interface and a structured control decomposition. That distinction explains both why the transfer works and where the method remains limited. Decoding the headline claims. “Single RGB camera” means a single monocular viewpoint, not a single isolated frame. The controller uses measurements from consecutive timesteps, including changes in box center, width, and height. It also stacks two consecutive policy observations. Motion is therefore inferred temporally, even though no depth channel or second camera is used. “Pixel-level features” also does not mean raw pixels or a learned visual embedding. The deployed actors receive a six-dimensional, control-oriented feature vector derived from the object’s image-space bounding box. The visual frontend performs segmentation; the reinforcement-learning policy never has to bridge the appearance gap between synthetic and real RGB imagery. Finally, “without explicit 3D position” applies to policy execution. During simulation training, the centralized value function receives privileged object state, and the rewards are computed using true object and robot-link positions. This is an asymmetric actor-critic setup: geometry is available to make training stable, but not to the deployed actors. The paper is therefore demonstrating 3D-free execution rather than 3D-free training. The multi-agent terminology deserves similar clarification. Pixel2Catch does not coordinate multiple robots. It factorizes one kinematic chain into two heterogeneous agents. The arm actor produces six actions and is rewarded primarily for interception. The hand actor produces 13 actions and is rewarded for bringing the palm and fingertips around the object. Both remain physically coupled through the robot and share visual and end-effector information. The physical system and task. The real platform consists of a six-degree-of-freedom Universal Robots UR5e mounted on a table 0.81 meters high. An Allegro Hand is attached to the wrist. Three hand joints are fixed, leaving 13 actuated joints under target-position control. A RealSense D435 observes the workspace from a fixed global viewpoint, approximately 0.5 meters behind and 2.2 meters above the robot. Although the sensor can produce depth, Pixel2Catch uses only its RGB stream. This gives the controller 19 action dimensions in total, divided into six arm and 13 hand dimensions. The camera is external rather than wrist-mounted, so it sees both the approaching object and the robot’s interception region. The released simulator uses a 640-by-480 pinhole camera with intrinsics matched to the real sensor. Policies send commands through ROS 2 at 30 hertz. The task is defined in two layers. A trial counts as tracked when the palm reaches the object, measuring arm interception quality. It counts as successful when the object remains captured rather than being dropped, measuring the combined arm-and-hand behavior. The released simulation code operationalizes these ideas as entering an eight-centimeter palm-to-object neighborhood at least once for tracking, and staying inside it for 20 consecutive policy steps for success. At 30 hertz, that hold requirement is roughly two-thirds of a second. That metric is worth remembering. The simulator’s success signal is a persistent proximity test, not a full contact-state or force-closure analysis. Real catches are evaluated behaviorally, but the training objective uses a tractable geometric proxy. From an RGB image to six control variables. The perceptual bottleneck is the core of Pixel2Catch. In the real system, SAM 2 segments the target object. A bounding box is placed around the segment, from which the system computes its center, width, and height. The policy representation is stated as:. current horizontal and vertical center coordinates;. changes in horizontal and vertical center coordinates;. changes in bounding-box width and height. This produces six features: current image location, apparent image velocity, and optical expansion or contraction. During training, the box corners are perturbed by as much as five pixels to model segmentation uncertainty. The horizontal and vertical center motion provide interception direction. If the object moves leftward in the camera image, for example, the policy learns an associated arm correction. Width and height changes provide a crude approach cue. An object growing rapidly in the image is generally getting closer, while slow growth suggests either greater distance or lower approach speed. This is not metric depth, and it is ambiguous under arbitrary changes in object size or orientation. But catching does not necessarily require a globally consistent depth estimate. It requires enough information to place the hand and decide when to close it. Optical expansion can serve as a learned time-to-contact signal even when the controller cannot recover an exact distance in meters. The released code makes the interface particularly clear. The current box center is normalized by image width and height, while temporal differences are computed from successive boxes. If segmentation temporarily produces no pixels, the simulator retains the previous bounding box rather than injecting an invalid measurement. The simulator itself obtains the object region through a simple green-color mask; SAM 2 is used only on the real stream. The control policy therefore sees the same compact feature definition in both domains even though the segmentation mechanisms are completely different. This is arguably the paper’s most transferable design principle. Instead of simulating photorealistic images or aligning high-dimensional visual embeddings, Pixel2Catch creates a narrow interface between perception and control. The simulator has to match projected object motion, not real-world appearance. SAM 2 handles appearance on the real side, while the RL policy consumes only geometry-like image statistics. There are two important qualifications. First, this pipeline assumes the target can already be identified and segmented. Open-world object discovery and target association are outside the problem formulation. Second, the paper does not report the SAM 2 model variant, prompt-initialization procedure, segmentation runtime, or end-to-end perception latency. Those details matter in a task where a few delayed frames can materially change interception. One robot, two agents. Pixel2Catch formulates the system as a multi-agent Markov decision process and trains it with MAPPO under centralized training and decentralized execution. The arm and hand have separate actors because their action dimensions, observations, and functional roles differ. During deployment, each actor generates commands independently. During training, centralized critic inputs provide information from the overall system and privileged simulator state. Per timestep, the arm actor receives 18 arm-specific quantities: six arm joint positions, six current joint targets, and six previously applied actions. It also receives 16 shared quantities: the seven-dimensional palm pose, three-dimensional palm displacement, and six visual features. That gives 34 values per frame, or 68 after stacking two frames. The hand actor receives 39 hand-specific quantities—13 joint positions, 13 targets, and 13 previous actions—plus the same 16 shared quantities. Its two-frame observation is therefore 110-dimensional. The centralized state adds the information from both subsystems and nine privileged quantities describing object position, object displacement, and hand–object relationships. The released configuration uses an 82-dimensional central state per frame and a 164-dimensional two-frame input. The arm and hand actions are interpreted differently. The arm policy outputs incremental joint-position commands that are added to the current positions and executed through a PD controller. Hand outputs are scaled to the Allegro joint limits and used directly as target joint positions. This asymmetry reflects the task decomposition: the arm continuously redirects the interception point, while the hand generates a grasp posture. The rewards give each actor a different learning signal while preserving shared terminal incentives. The arm receives progress for decreasing palm-to-object distance, a small approach bonus, a large catch bonus, penalties for dropping or collision, a time cost, and an action-magnitude penalty. The hand receives the average distance progress of the palm and four fingertips, plus the shared catch and failure terms and its own action penalty. The published weights are 10 for success, 5 for failure, 0.1 for approach, 0.01 for action magnitude, and a time cost of 0.01 per step. This shaping creates a useful compromise. The agents are not trained on completely independent objectives: both care about the final catch. But the dense terms tell the arm to solve interception and the hand to solve enclosure. A single policy instead has to discover this factorization implicitly while processing all joint states and balancing one combined reward. Both actors and critics are multilayer perceptrons with hidden widths of 512, 256, and 128 and ELU activations. The paper reports a PPO clipping ratio of 0.2, discount factor of 0.99, and adaptive KL threshold of 0.016. The released configuration adds a learning rate of (5\times10^{-4}), generalized-advantage parameter of 0.95, 32-step rollouts, four learning epochs, four minibatches, and 100,000 vectorized trainer steps. Training uses 512 parallel environments on two RTX A6000 GPUs. Building the sim-to-real bridge. The Isaac Lab environment reproduces the UR5e, Allegro Hand, table, and camera geometry. Physics runs at 120 hertz, while each selected action is repeated for four physics steps, yielding the same 30-hertz command rate used on the real robot. Episodes last up to three seconds. Five primitive object geometries are used during training, with their scales varied to increase shape and size diversity. Additional geometries are reserved for simulated validation. Initial object position, orientation, linear velocity, mass, and throw direction are randomized. In the released environment’s coordinate system, objects begin roughly 2.4 to 2.9 meters along the throwing axis, within 0.3 meters laterally and near 0.8 meters high. Target interception regions vary laterally and span approximately 0.9 to 1.1 meters in height. The dynamics gap is attacked in two stages. First, the authors execute diverse joint-space trajectories on the real robot, record the response, and tune simulation parameters such as actuation gains, damping, friction, and armature to minimize tracking error. Second, they randomize around that identified model. Arm stiffness and damping are scaled between 0.8 and 1.2, hand stiffness and damping between 0.7 and 1.3, object mass between 0.5 and 1.5 times nominal, and restitution between zero and 0.5. Action noise, observation noise, and randomized initial joints are also applied. The resulting transfer recipe has four mutually reinforcing parts: match the control frequency, identify the nominal robot dynamics, randomize around the remaining uncertainty, and avoid exposing the actor to high-dimensional synthetic images. None of these elements is individually novel, but the combination is well suited to this task. One reproducibility gap is that the paper does not provide the identified actuator parameters, the optimization method used to fit them, or before-and-after trajectory errors. The public code releases the simulation task, assets, MAPPO configuration, and training commands, but reproducing the exact real robot still requires redoing system identification locally. Results: what actually moved the needle. A note on paper versions. The public artifacts contain two generations of the experiment. The arXiv v1 was posted on February 26, 2026, before the paper’s May acceptance to RA-L. It reports 30 real trials per object and does not include the 3D-position baseline. The author-maintained project page accompanying the journal publication reports an expanded evaluation of 360 trials per policy: three objects, two human throwers, two backgrounds, and 30 trials per combination. It also adds a 3D-position baseline. The discussion below uses that expanded result set while distinguishing it from the earlier preprint. Simulation performance. The simulation evaluation covers both training geometries and unseen geometries. Each policy is tested with 10 trials in each of 100 parallel environments, and learning results are averaged over three random seeds. The table below lists the mean rates reported on the updated project page. | Method | Seen tracking | Seen success | Unseen tracking | Unseen success | |---|---:|---:|---:|---:| | No pixel features | 12.13% | 8.93% | 12.11% | 7.89% | | Width/height only | 8.03% | 5.53% | 6.11% | 4.00% | | Center only | 87.07% | 81.27% | 87.50% | 80.72% | | Single-agent RL | 78.20% | 63.50% | 75.83% | 65.44% | | Oracle 3D position | 93.33% | 90.43% | 91.83% | 89.10% | | Pixel2Catch | 89.97% | 84.13% | 89.28% | 84.83% |. Several points stand out. Without continuous visual motion information, performance is essentially nonexistent. Initial throw state and proprioception cannot compensate for trajectory variation. Width and height alone are equally inadequate because they do not tell the arm where to move laterally or vertically. Center-only observations are surprisingly effective in simulation. They reach more than 80% success because the simulated trajectory and object distributions are sufficiently regular for image-plane direction to carry most of the task. Adding scale-change cues improves success by only about three to four percentage points in simulation. The multi-agent comparison is much larger. Pixel2Catch improves success over the single-agent policy by 20.6 percentage points on seen objects and 19.4 points on unseen objects. It also shows almost no degradation from seen to unseen geometry. This supports the claim that role-specific actors and rewards make the high-dimensional controller substantially easier to optimize. Oracle 3D position remains best in simulation, which is exactly what one should expect. Metric state contains more direct information when it is perfectly accurate. Pixel2Catch trails it by 6.3 points of success on seen objects and 4.3 points on unseen objects. The paper’s case is therefore not that image-space features are intrinsically superior, but that they retain most of the useful control information while offering a cleaner path across the reality gap. Expanded real-world performance. The expanded real evaluation uses three object geometries—cube, L-block, and triangle—and varies both thrower and background. The compact table below shows tracking and success as ; the final column is the mean calculated across the three objects. | Method | Cube | L-block | Triangle | Mean | |---|---:|---:|---:|---:| | Width/height only | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | | Center only | 44.73 / 15.27 | 44.80 / 10.60 | 47.47 / 17.27 | 45.67 / 14.38 | | Single-agent RL | 64.67 / 31.27 | 57.27 / 34.00 | 61.40 / 30.00 | 61.11 / 31.76 | | 3D position | 70.83 / 45.83 | 65.83 / 44.17 | 70.00 / 35.83 | 68.89 / 41.94 | | Pixel2Catch | 77.93 / 60.60 | 76.60 / 51.27 | 74.67 / 46.60 | 76.40 / 52.82 |. Pixel2Catch averages 76.4% tracking and 52.8% catching success. It improves success over single-agent RL by just over 21 percentage points and over the reported 3D-position baseline by almost 11 points. The earlier arXiv evaluation found approximately 72% mean tracking and 50% mean success, so the expanded results preserve the original conclusion while testing more thrower and background combinations. The tracking numbers are relatively consistent across geometry, ranging from 74.7% to 77.9%. Success varies more: 60.6% for the cube, 51.3% for the L-block, and 46.6% for the triangle. That pattern suggests that the arm policy generalizes better than the final grasp. Once the palm reaches the object, hand closure and impact retention remain sensitive to shape. There is also a 23.6-point gap between mean tracking and mean success. Pixel2Catch solves much of the interception problem, but roughly one-third of palm–object encounters still fail to become retained catches. This is the clearest indication of where tactile sensing, impact-aware control, or richer hand-state estimation could help. What the ablations really establish. The center-only result is the most revealing ablation. It is nearly competitive in simulation but collapses to 14.4% mean real-world success. The arm can often move toward the correct image location, yet the hand lacks a robust cue for approach timing. In the preprint, the authors describe oscillatory finger behavior and eventual drops when scale information is absent. Width and height change appear less important for nominal simulated trajectories than for out-of-distribution timing and transfer. The width/height-only policy fails everywhere because optical expansion does not specify interception direction. This confirms that the cues are complementary rather than interchangeable: center motion primarily answers “where,” while scale change helps answer “when.”. The multi-agent result supports structured control decomposition, but it does not isolate MAPPO as the only cause. The single-agent baseline differs in policy factorization, observation coupling, reward structure, and likely optimization geometry. The paper does not report a parameter-matched baseline with two independently parameterized actors trained without a centralized critic, nor a single actor with explicitly separated reward heads. Consequently, the evidence is strong for modularization as a package, but weaker for attributing the entire gain to the multi-agent algorithm itself. The reported 3D baseline is also instructive but should be interpreted cautiously. The project page explicitly describes its simulation input as oracle coordinates. It provides less implementation detail about how corresponding real-world 3D measurements are produced. Its real-world underperformance is consistent with the paper’s representation-mismatch argument, but it is not enough to conclude that monocular cues will generally outperform a carefully engineered, low-latency 3D tracker. Where Pixel2Catch sits among catching systems. The closest recent comparison is Catch It! Learning to Catch in Flight with Mobile Dexterous Hands. That system combines a mobile base, six-degree-of-freedom arm, and 12-degree-of-freedom hand, and uses a two-stage reinforcement-learning pipeline. It reports roughly 80% simulated success and transfers to real catches of human-thrown sandbags. Its mobile base provides a much larger interception workspace, while Pixel2Catch focuses on simplifying perception and replacing staged policy learning with simultaneous heterogeneous-agent training. At the other extreme, Event-based Agile Object Catching with a Quadrupedal Robot attacks sensing bandwidth directly. Its event camera and 100-hertz pipeline enable catches of objects moving as fast as 15 meters per second, with an 83% reported success rate. Pixel2Catch operates in a different regime: ordinary RGB at 30 hertz, lower-cost sensing, and active multi-finger grasping rather than high-speed interception alone. Other systems cited by Pixel2Catch use motion capture, RGB-D reconstruction, ballistic prediction, model-predictive control, passive baskets or scoops, and multi-stage dexterous policies. There are also prior modular hand–arm and heterogeneous-agent catching methods. Pixel2Catch’s novelty is therefore not the isolated invention of monocular cues, modular control, or MARL. Its contribution is their particular integration: a fixed-camera image-space interface, one-stage arm–hand CTDE, and direct transfer to a 19-DoF physical system. Why the paper matters beyond catching. The strongest lesson is about representation design. A common sim-to-real instinct is to improve realism until a high-dimensional vision policy survives deployment. Pixel2Catch takes the opposite route: remove visual information that the controller does not need. Once appearance is reduced to center motion and scale change, synthetic-to-real visual alignment becomes almost trivial. This approach is particularly attractive for known-target dynamic manipulation. Conveyor interception, handovers, batting, catching parts ejected from machinery, and aerial docking all contain situations where image-plane motion may be more reliable than noisy metric reconstruction. A policy may not require a globally consistent world model if it can react to task-sufficient projective cues. The work also illustrates the value of treating a complex robot as a society of functional modules rather than a single monolithic actor. Arm motion and finger closure happen on the same machine, but their control objectives, action scales, and relevant proprioception differ. Encoding that structure into the learning problem reduces the burden on optimization. Finally, Pixel2Catch provides a useful example of privileged training done correctly. The critic and reward system can exploit exact simulator geometry without forcing the actor to depend on it. The simulator is used as a teacher for control, not as a source of observations that must somehow be reconstructed perfectly in reality. Limitations and open questions. The first limitation is camera placement. A fixed external camera offers a stable viewpoint, broad visibility, and relatively simple segmentation. It does not test moving-camera geometry, self-occlusion, hand-mounted sensing, or deployment in an uninstrumented environment. Extending the method to a mobile manipulator would require either camera-motion compensation or a policy trained across changing extrinsics. Second, segmentation is treated as an available service. SAM 2 is powerful, but the paper does not quantify segmentation failures, initialization effort, frame rate, or latency. The policy representation also discards uncertainty: a noisy box is still passed as six deterministic numbers. A practical successor should expose segmentation confidence, maintain a belief over box motion, and explicitly train with dropped or delayed detections. Third, the real evaluation remains narrow despite the welcome expansion to two throwers and two backgrounds. It includes three labeled geometries but does not systematically vary mass, rigidity, surface friction, texture, rotational velocity, or throw speed. The simulator randomizes some of these properties, yet no real-world breakdown tells us which variations dominate failure. Fourth, the hand receives no tactile input. Its closure timing and retention decisions remain visual and proprioceptive. Once the object is partially occluded by the fingers, image-space scale becomes less informative precisely when contact feedback would be most useful. The tracking-to-success gap suggests that a tactile residual policy could deliver a substantial improvement without changing the arm controller. Fifth, the paper’s terminology around width and height is not perfectly consistent. The formal six-dimensional definition and released code use current center, center displacement, and changes in width and height, while some project-page descriptions speak more generally of width, height, and their temporal differences. This does not undermine the result, but exact feature construction matters for reproduction. Finally, system identification is crucial but underdocumented. Researchers adapting the code to a different arm or hand will need their own excitation trajectories, actuator fitting procedure, latency measurement, and safety validation. The released simulation code is valuable, but it is not a turnkey real-robot stack. How researchers might build on it. A natural next step is to preserve Pixel2Catch’s compact perception interface while making it explicitly sequence-aware. A small recurrent model could consume a longer history of center, scale, and confidence, estimating curvature and time-to-contact without reconstructing full 3D trajectories. The modular policy structure also invites targeted upgrades:. Add tactile or motor-current observations only to the hand agent, leaving the transferred arm policy unchanged. Train an impact-aware hand critic that rewards relative-velocity reduction and object retention, not just persistent palm proximity. Randomize camera pose and introduce ego-motion features for wrist-mounted or mobile deployment. Replace a single deterministic bounding box with a distribution over object extent and image velocity. Introduce a third timing or coordination agent that gates aggressive arm motion and hand closure without returning to a fully monolithic controller. The open-source release makes these experiments unusually accessible. It includes the Isaac Lab task, robot and table assets, procedural thrown objects, MAPPO configuration, and reproduction commands. That should make Pixel2Catch useful not only as a catching system but also as a testbed for heterogeneous control, uncertainty-aware visual servoing, and modular sim-to-real research. Final assessment. Pixel2Catch does not solve general robotic catching, and it does not eliminate geometry from the training process. What it demonstrates is narrower and more useful: a dexterous robot can learn a substantial catching skill without metric 3D observations at deployment, provided that the visual representation preserves direction and optical expansion, the arm and hand receive appropriately specialized objectives, and the dynamics pipeline is engineered for transfer. Its roughly 76% real-world tracking rate and 53% catch rate leave considerable room for improvement. Yet the ablations make the core result convincing. Center motion gets the robot to the object; scale change makes the timing transfer; heterogeneous policies turn more encounters into stable grasps. The broader message is that sim-to-real performance often depends less on making simulation look real than on choosing an observation space in which simulation and reality already look alike.