Latest / Embodied AI 101 / PEAOM: Better Sim-to-Real Table Tennis by Modeling Camera Errors
Transcript
- PEAOM: Better Sim-to-Real Table Tennis by Modeling Camera Errors. Perception-Error-Aware Observation Modeling for Zero-Shot Sim-to-Real Control of Robotic Table Tennis • Hongxu Ma, Jianyin Fan, Haoran Xu, Qiang Wang • Harbin Institute of Technology • International Journal of Control, Automation, and Systems • 2026. Cold Open. The robot has learned the shot. In simulation, its racket arrives where the ball will be, with the right speed and angle. On hardware, it swings past. Now imagine that the motors did exactly what they were asked to do. The error started earlier: the cameras displaced the measured trajectory, the trajectory fitter changed its velocity estimate, and the controller scheduled the strike from that distorted state. The miss began upstream of the policy. This is a meaningful distinction in robotic table tennis. Earlier systems research found that policies could tolerate substantial zero-mean observation noise while suffering much more from biased measurements. Random jitter and systematic error created different control problems. PEAOM—perception-error-aware observation modeling—targets that distinction. Ma and colleagues train a striker using simulated measurements shaped like their physical vision system’s output. The question is which measurement errors simulation must reproduce to teach the right physical response. The Idea. Start at the interface between perception and control. The released implementation passes timestamped ball positions into a trajectory-fitting controller. It fits quadratic motion, derives position and velocity, and predicts an interception. The actor receives estimated ball position and velocity, a desired landing location, and a desired return speed. Its outputs specify racket position, orientation adjustments, a timing parameter, and forward racket speed. These parameters describe one stroke: where to meet the ball, how to orient the paddle, and when to arrive. PEAOM changes how the measurements entering that pipeline are generated. The authors’ calibration code fits an affine mapping from reference ball coordinates to binocularly triangulated coordinates. In plain language, it mixes the three spatial coordinates and adds an offset. That can express translation errors, scale errors, and cross-axis coupling—for example, a measured-height error that changes with depth. The fitting procedure uses paired positions and ordinary least squares. Its direction is deliberately from physical reference position to reported visual position: exactly what a simulator needs to imitate a measurement system. Every simulated trajectory sample passes through that mapping before state estimation. Imagine a depth-dependent height distortion. As the ball moves, its apparent height changes systematically; the fitted vertical velocity changes with it. A prediction of when the ball reaches the striking region can also shift. The released sensitivity-analysis code explicitly evaluates this propagation into position, velocity, and predicted hitting time. The important object is the entire estimated trajectory, with linked errors across its components. Notice what remains physically consistent. The simulated ball continues along its physical trajectory, and the racket must contact that ball. The observation model changes the position stream sent to the controller. Training therefore exposes the actor to imperfect information about a coherent physical world, rather than relocating the physical ball to wherever the camera appears to see it. This is also a representation-level choice. PEAOM emulates three-dimensional measurements rather than rendering camera images and recreating every stage of visual processing. Its operational interface is the ball-position stream already consumed by the estimator. The calibration model is compact because it operates at that interface. Downstream learning uses a single-step, entropy-regularized actor–critic formulation. The released trainer stores state, action, and outcome triples. Two critics learn to predict stroke-outcome components, and the actor searches for low-cost strokes while retaining exploration. Training initially emphasizes contact alignment, then includes the complete return objectives. The learning unit is a parameterized physical event—a strike—with simulated interactions supplying experience. The training data consequently have distinct roles. Paired calibration positions identify the observation mapping. Randomized simulated launches supply control experience by varying incoming positions and velocities. For physical evaluation, the configuration selects the trained actor without changing the physical camera trajectory, and real-world policy fine-tuning is disabled. That is the zero-shot contract here: calibrate the observation process, learn in simulation, and deploy the actor unchanged. There is useful precedent beyond table tennis. In Champion-level drone racing using deep reinforcement learning, Kaufmann and colleagues modeled real odometry residuals with Gaussian processes and sampled temporally consistent errors during simulated policy refinement. PEAOM belongs to that sensor-aware sim-to-real tradition. Its concrete engineering contribution is the integration of a calibrated measurement model with the downstream trajectory estimator used by a robotic striker. The Result. The headline is a reported physical return-success rate rising from 56.4% to 86.0%, comparing ideal-observation training with PEAOM. The authors also report improved hit rate, fewer missed-ball and contact-related failures, and smaller landing errors among successful returns. These are physical-platform results obtained without policy fine-tuning. Return success measures whether the attempt produces a successful return; landing accuracy is a separate, conditional measurement on returns that succeeded. The useful experimental unit is the observation model. The released configuration selects separate ideal, Gaussian, and PEAOM training conditions while retaining a shared controller and timing interpretation. During physical evaluation, that selection determines which trained policy is loaded. It does not inject synthetic errors into the real camera measurements. This makes the comparison specifically about how simulated observation generation prepares the policy for deployment. The Gaussian comparison deserves careful attention. In the simulator, Gaussian noise is also applied to trajectory samples before fitting. The controlled contrast is independent, zero-mean perturbation versus a calibrated systematic distortion, both processed downstream by the estimator. Baseline PEAOM uses the deterministic affine mapping; an additional mode combines that mapping with Gaussian residuals. The central variable is the structure of the measurement error. The repository describes a revealing additional ablation: Gaussian noise scaled to the raw, uncompensated measurement error did not produce a usable policy in the reported high-noise setting. That supports an engineering interpretation: matching total error magnitude is different from matching how errors arise. A coherent displacement of a trajectory and independent scatter around every point can produce comparable aggregate error while presenting very different learning problems. Earlier table-tennis experiments make this interpretation plausible. D’Ambrosio and colleagues’ Robotic Table Tennis: A Case Study into a High Speed Learning System, published at RSS in 2023, explicitly distinguished tolerance to zero-mean noise from sensitivity to bias. Their observation interpolation and smoothing helped suppress jitter; systematic offsets remained more damaging. My technical takeaway from the comparison is that the observation model should preserve the relationship between measurements and the estimates derived from them. “Add more noise” treats uncertainty as a magnitude. PEAOM treats a repeatable distortion as a mechanism that the simulator can reproduce. That is the result’s most useful connection to the broader sim-to-real literature. The Biggest Weakness. The biggest unresolved comparison is against using the same calibration to correct observations before controlling the robot. A source boundary matters here: I could inspect the publisher’s public abstract and the official implementation, but not the subscription-only article body. The publicly documented comparison suite includes ideal, Gaussian, and PEAOM variants; it does not document an inverse-calibration control baseline. This is an unanswered question in the evidence available for this episode, rather than a claim about every experiment in the full text. That baseline is unusually natural. The method already estimates an affine mapping from reference coordinates to measured coordinates. When that mapping is invertible, one can subtract the offset and undo the coordinate mixing on incoming camera measurements, then pass the corrected samples through the existing estimator. An ideal-observation-trained policy would receive estimates closer to its training inputs. The released analysis utility already performs this inverse operation when estimating whether physical trajectory samples lie inside the calibrated workspace. That is not a control experiment, but it establishes a concrete implementation route for the comparison. The distinction separates two explanations for the improvement. One is that learning with structured measurement errors provides an advantage beyond recalibration. The other is that PEAOM learns to compensate for known bias that remains in the comparison policies’ inputs. The public experiments distinguish systematic structure from independent jitter; they do not resolve this second question. An inverse map can amplify residual noise, so training-based compensation could still win. That is precisely why the experiment matters. Use the same calibration data, estimator, physical evaluation protocol, and policy-training budget. Compare PEAOM against an ideal-trained policy receiving corrected measurements. That would tell practitioners why the compensation should live inside policy training rather than at the perception interface. Verdict. PEAOM deserves attention from teams building vision-guided interception systems. Its useful design principle is to treat the measurement-and-estimation pipeline as part of the environment against which the controller learns. For catching, striking, or another timed-contact task, I would start by asking which intermediate estimates actually determine the action. Those are the representations whose error structure needs to survive the trip through simulation. Sony’s Ace provides a complementary view of the field. In the 2026 paper Outplaying elite table tennis players with an autonomous robot, Ace combines conventional cameras for ball position with event-based vision for spin estimation. Its asymmetric actor–critic training gives the critic true simulated ball states while the actor receives noisy measurement histories. That system illustrates how tightly perception design and policy training can be integrated in competitive robotic table tennis. PEAOM focuses attention on a compact, explicit observation emulator at that same interface. For implementation-oriented readers, the authors’ repository contains a PyBullet environment, calibration utilities, training and evaluation code, and reference actor checkpoints. That makes the central mechanism inspectable as software: a measurement transformation feeding an estimator and a learned striker. The broader research lesson is more precise than “make simulation realistic.” Identify the stage where your hardware produces measurements, and preserve the computations that turn those measurements into decisions. This is a task-dependent definition of fidelity. For a state-based controller, a compact model of structured measurement error can represent information that a visually elaborate simulator never supplies to the policy. That last point is an engineering inference from PEAOM’s design, and it is the reason to read the work. Its value lies in connecting calibration, estimation, and policy training around a specific control event. The observation model becomes a testable component of the learning system. The takeaway: train against the observation process the robot will actually use. Outro. Think back to the racket that swung past the ball. From the controller’s perspective, it may have performed the right action for the trajectory it was given. That is the relationship worth keeping in view: physical motion produces measurements; measurements produce estimates; estimates produce commands. Improving any one stage becomes more useful when we understand how its errors travel through the others. Watch for observation emulators to become reusable components of robot-learning systems, with calibration and state estimation evaluated together. You’ve been listening to Embodied AI 101. The ball follows physics. The robot follows an estimate. Good sim-to-real engineering connects both.