Latest / Embodied AI 101 / FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Control
Transcript
- FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Control. Reinforcement learning (RL) has made impressive strides on many tasks, but controlling high‐dimensional humanoid robots remains a major frontier. In practice, state-of-the-art controllers for 3D humanoids have often leaned on deterministic or near-deterministic policies (e.g., variants of TD3 or PPO) rather than fully stochastic agents. The culprit is the “curse of dimensionality”: with dozens of joints and actuators, a naive Gaussian policy wastes its exploration budget on irrelevant degrees of freedom, leading to vanishing exploration and unstable training. Maximum-entropy RL methods – like Soft Actor-Critic (SAC) – promise powerful stochastic policies and robustness, but in practice they have underperformed on complex humanoid tasks, losing ground to tuned deterministic methods. The new FastDSAC framework (Xue et al., 2026) challenges this trend. FastDSAC is a variant of off-policy maximum-entropy RL specifically designed for high-dimensional humanoid control. The key idea is to redistribute the exploration noise across joints and to use a distributional critic that models return uncertainty. Concretely, FastDSAC introduces Dimension-wise Entropy Modulation (DEM) – a learned per-joint scaling of the action variance – so that the policy can focus its stochasticity where it matters. At the same time, the value function is learned distributionally as a Gaussian (instead of a single expected value), improving value estimates in large action spaces. Integrated into a “Distributional Soft Policy Iteration” algorithm, these innovations allow a Sac-like agent to finally tame humanoid control. Empirically, FastDSAC delivers striking results. On a suite of 39 simulated humanoid tasks (spanning locomotion, manipulation, and coordination from the recent HumanoidBench and similar benchmarks), FastDSAC matches or surpasses the best baselines on every task. In fact, it achieves 180% higher return than a tuned deterministic baseline on a simulated basketball throw, and 400% higher on a “Balance Hard” task (balancing on a sweeping ball). Remarkably, FastDSAC often learns qualitatively different strategies – for example, using its torso as a rebound surface to deflect a flying basketball rather than trying (and failing) to catch it with an unstable hand. In short, FastDSAC transforms “noise” from a nuisance into a resource, opening the door to powerful stochastic control policies in domains that previously seemed prohibitive for maximum-entropy RL. The Challenges of High-Dimensional Humanoid Control. Humanoid robots have dozens of joints and actuators. Learning even simple behaviors (walking, jumping, reaching) requires coordinating many degrees of freedom. Classic RL struggles here because random exploration in a high-dimensional space is extremely inefficient. Concretely, a standard SAC actor outputs independent Gaussian noise on each joint. With, say, 20+ joints, drawing a typical random action will almost never produce any coherent whole-body motion – legs might flail in random directions, arms drift aimlessly, and so on. Worse, much of that noise will occur in joints that are not even relevant to the current task, effectively wasting the exploration budget. This phenomenon has been called exploration collapse: as training proceeds, the policy may collapse to being nearly deterministic in practice, because spreading noise thinly across all dimensions either fails or becomes counterproductive. Practitioners have mostly sidestepped this by using deterministic policy gradient methods (TD3, DDPG, etc.) for humanoids, or heavy parallelism with modified update rules. While those can work, they give up the benefits of explicit exploration and stochasticity that max-entropy RL offers. In theory, a properly managed stochastic policy could discover richer behaviors and be more robust to perturbations. In practice, prior high-throughput methods have opted for tricks like truncating or clamping actions, clipping gradients, or simply ignoring the entropy bonus in the objective. For example, a recent parallelized SAC variant (“FastDSAC” by Lu et al., 2026) resorted to using a truncated Gaussian policy to eliminate extreme outliers during massive batch updates. But these fixes only manage symptoms. Instead, FastDSAC takes a different perspective: it fully embraces the maximum-entropy framework, but rethinks how to allocate entropy when actions are high-dimensional. The core insight is that the policy should learn to decide which joints get noisy and which stay quiet, rather than treating all joints equally. In high-dimensional action spaces, a policy that can redistribute its total exploration budget moment-to-moment can remain stochastic where needed (to discover new strategies) and reduce randomness elsewhere (to maintain stability). FastDSAC operationalizes this with Dimension-wise Entropy Modulation, as described next. Dimension-wise Entropy Modulation (DEM). At the heart of FastDSAC is Dimension-wise Entropy Modulation (DEM): a mechanism that lets the agent dynamically reallocate how much exploration noise it injects into each action dimension. In a standard SAC, the policy over actions is a multi-dimensional diagonal Gaussian. Each joint (i) has a learned mean (\mu_i(s)) and a standard deviation (\sigma_i(s)) (or log-standard-deviation output by the network). FastDSAC augments this by also producing modulation weights (w_i(s)) for each joint. These weights adjust the per-joint noise such that not all joints get equal baking in exploration. Concretely, the actor network outputs two things for each action dimension (i): a base log-variance (\hat\sigma_i(s)), and a raw logit (l_i(s)). The logits (l_i) are passed through a temperature-scaled Softmax (with a diversity factor (\beta_e) and temperature (\tau)) across all dimensions. This yields normalized weights (w_i\ge 0) such that[ w_i ;=; N;\frac{\exp(l_i \beta_e/\tau)}{\sum_j\exp(l_j \beta_e/\tau)},, ] where (N) is the number of action dimensions. By construction, the average weight is 1; ((\tfrac1N\sum_i w_i = 1)), enforcing a variance-conservation constraint. The final standard deviation for joint (i) is then[ \sigma_i(s) ;=; \exp(\hat\sigma_i(s)) \cdot w_i(s). ]In other words, each joint’s base noise is scaled by its weight (w_i). If (w_i<1), that joint’s exploration is suppressed; if (w_i>1), it is amplified, while ensuring the total exploration budget remains fixed. This weight-based mechanism has a profound effect. It allows the agent to autonomously “prune” subspaces of the action space. Intuitively, if a particular joint is irrelevant to the reward at a given moment, the policy can learn to set (w_i) near zero, effectively turning off exploration there. The saved noise is then redistributed to other joints via the Softmax normalization. As the authors note, this “structural constraint” lets the agent take its total exploration budget and reassign it across dimensions. The result is that important joints get more randomness, and unimportant ones get less, rather than naively spreading noise everywhere. Importantly, the total entropy of the policy is kept roughly constant – the policy is still a maximum-entropy policy – but the allocation of that entropy is learned. The paper’s authors emphasize that this subsystem enables highly targeted exploration. In practice, they found that DEM encourages a very intuitive effect: the policy concentrates noise into a few “sink” joints and quiets the rest. For example, imagine a humanoid reaching for an object: DEM might direct most of the exploration into the wrist joint that guides the gripper, while freezing the torso and supporting leg to ensure stability. In a sense, one joint can become an “entropy sink” that absorbs randomness to protect the rest of the body. As we will see in the experiments, this mechanism is crucial. In ablation studies, removing DEM (having all (w_i=1) as a fixed uniform weight) causes learning to collapse: performance plunges and training becomes unstable. FastDSAC’s performance gains hinge on this ability to allocate exploration flexibly. Continuous Distributional Critic. FastDSAC’s other major innovation is in the critic – the value function network. Instead of predicting a single expected return (Q(s,a)), FastDSAC’s critic predicts a full return distribution, parameterized as a Gaussian. Concretely, each critic head outputs two values: a mean (Q_\psi(s,a)) and a variance (\sigma^2_\psi(s,a)). We interpret these as modeling the random return (Z_\psi(s,a)\sim\mathcal{N}\bigl(Q_\psi(s,a),,\sigma^2_\psi(s,a)\bigr)). Why go distributional? Distributional RL (e.g. C51 or QR-DQN) has shown that learning about the distribution of future returns, rather than just the mean, can improve stability and accuracy of value estimates. In FastDSAC’s context, modeling the return as a continuous Gaussian brings two benefits. First, it avoids the quantization errors of fixed-bin schemes (like C51) that may struggle to model fine-grained value differences. As the FastDSAC authors explain, discrete bins introduce small inaccuracies (“quantization error”) that accumulate and prevent the very precise control needed for complex humanoid tasks. A continuous parameterization can capture subtle changes in returns more faithfully. Second, by explicitly learning the uncertainty (the (\sigma^2)), the critic can adjust learning updates according to the confidence in its predictions. FastDSAC’s distributional critic is also simpler to implement than fancy discrete schemes. The Gaussian form means we only need to minimize a divergence between predicted and target Gaussians. In practice, the authors use a variant of the soft Bellman update: the target return (r + \gamma Z_\psi(s',a')) is also Gaussian (since the sum of a constant and a normal is normal), and the critic is trained (via a KL or (\ell_2) loss) to match that target Gaussian. Crucially, the authors dispense with any handcrafted clipping of variance and instead rely on the stability afforded by large-batch training. They report that with their parallel simulator setup, gradients remain well-behaved even with very small variances (they had to reduce the numerical floor from 0.1 to (10^{-6}) to allow very low-variance predictions). The upshot is a critic that provides higher-fidelity value signals. By fitting a Gaussian distribution, the critic can “propagate the value signal with higher fidelity” than a single-number critic. In other words, the learned value mean and variance can adapt to the shape of rewards in a way that discrete bins or single heads cannot. This helps mitigate one of the classic problems in off-policy critic learning: overestimation. If the critic is uncertain (high variance), it will be more conservative in updating, and vice versa. The paper reports that this continuous distributional approach greatly reduces the usual value overestimation that can plague standard SAC, especially in high-dimensional action spaces where random combinations can produce spurious high-Q estimates. Combined with the actor-solftware, this yields a novel Distributional Soft Policy Iteration scheme: the actor and (distributional) critic are updated in tandem using the familiar off-policy SAC formulas, but with the above extensions. Just like SAC, FastDSAC still optimizes the expected Q-value plus an entropy bonus, but now both the policy and the critic are richer. The policy samples actions from the modulated Gaussian, the critic produces a full Gaussian target, and the Bellman backup is distributional. The authors do not spell out every algorithmic detail in prose, but the effect is clear: FastDSAC inherits the convergence and efficiency of SAC while counteracting its failure modes in large action spaces. Scaling Stochastic Policies: Results on HumanoidBench. The authors evaluate FastDSAC on a broad custom benchmark of humanoid tasks. Roughly speaking, they consider 39 tasks drawn from modern suites like HumanoidBench (Lin et al., SciRobot 2024) as well as some MuJoCo Playground variants and Isaac Lab tasks. These tasks range from straightforward locomotion (running, walking on flat or uneven ground, bounding, hurdles) to highly complex full-body coordination tasks – for example, throwing and catching a basketball, balancing on a rolling sphere, multi-limb manipulation puzzles, and so on. Against this battery, FastDSAC is a clear winner. Across the board it matches or surpasses the best baselines. The baselines include a tuned fast TD3 variant (essentially a high-throughput deterministic policy) and the vanilla SAC. In aggregate, FastDSAC is on par with baselines on easy tasks, and significantly better on complex ones. For instance, on the coordinated Basketball task, FastDSAC’s final return is about 180% higher than the FastTD3 baseline, which often fails to even complete a reliable throw. In the Balance Hard task (a humanoid balancing on a sweeping ball), the margin is even more dramatic: FastDSAC scores about 500% (i.e. five times) of the baseline’s return. On simpler locomotion tasks like running or hurdling, FastDSAC initially learns a bit more slowly, but eventually converges to a better controller than the baselines (smoother gait, higher running speed). Figure 2 in the paper (reproduced from the results) shows the final returns for each task: FastDSAC’s bars dominate on difficult tasks, and are never worse on any task. The text summary makes this clear: “FastDSAC achieves ~180% higher return than FastTD3 on Basketball, and 400% higher on Balance Hard… Across 39 diverse tasks in HumanoidBench, MuJoCo Playground, and IsaacLab, FastDSAC consistently matches or exceeds state-of-the-art baselines”. These are very large improvements, not incremental. The high-dimensional tasks had long been thought too difficult for a fully stochastic agent, but FastDSAC literally “unlocks” performance that deterministic policies were missing. It also converges much faster on these hard tasks, since the focused exploration finds the solution sooner. Beyond raw scores, the authors performed ablation studies. Tellingly, if they remove the DEM mechanism (using fixed noise scale across all dims instead), learning breaks down. Without DEM, FastDSAC behaves essentially like vanilla SAC in a big space – the training curves degrade dramatically and performance struggles to rise. This confirms that the dimension-wise modulation is essential: it is the lever that allows entropy to be used effectively in high-D control. Emergent Strategies: “Entropy Sinks” and Body Rebounds. One striking outcome of the experiments is that FastDSAC doesn’t just win on scores – it discovers qualitatively different behaviors. A highlight example is from the basketball-throwing task. The deterministic baseline or standard SAC often tries to catch the ball with its palms and misses balance. FastDSAC, by contrast, learns to redirect the ball with its torso in one fluid motion. The agent deliberately props its legs rigidly and ducks its torso under the incoming ball, using the torso as a trampoline to bounce the ball toward the hoop. This “body rebound” strategy is counterintuitive but robust: by not even attempting to palm-catch (which would almost surely tip it over), the humanoid stays upright and still makes the shot. How does FastDSAC find this behavior? The paper’s analysis turns to the learned DEM weights during a throw. They visualize the weight (w_i) for each joint over time. Remarkably, just at the moment of ball impact, FastDSAC quenches all noise in the large joints and channels it into a small finger joint. In the cited example, the weights on both legs and the torso drop nearly to zero at impact, while the weight on the left thumb jumps to around 8.0. In effect, the left thumb becomes an “entropy sink” – it absorbs virtually all the exploration noise, shielding the rest of the body from randomness. The agent has learned: “I can’t risk any disturbance in my legs or hips when the ball hits, so I’ll dump my randomness into a harmless fingertip.” This is exactly what DEM was designed to allow. This emergent strategy is precisely what the authors describe: “the body rebound strategy”. Instead of foolishly trying to catch with the hands (sacrificing balance), the agent holds a fixed stance and uses its torso to ricochet the ball. The coordinated noise burst in the thumb, while odd-sounding, actually stabilizes the rest of the body. The authors note: “This counter-intuitive but effective strategy emerges from FastDSAC’s entropy modulation”. Thus, FastDSAC not only achieves higher scores; it demonstrates that a fully-stochastic policy can use randomness in smart ways. In essence, “noise” is no longer just random dithering – it’s a resource that the policy invests where needed and contains where dangerous. The paper even shows a heatmap (Figure 3) of the (w_i) weights over time, highlighting exactly this effect: at impact, the torso joints (leg and trunk) are clamped close to zero, while one fingertip’s weight spikes as the entropy sink. This kind of analysis is possible only because the DEM mechanism gives clear semantic meaning to each weight. In summary, FastDSAC reveals that by properly shaping entropy, an agent can learn sophisticated behaviors that deterministic methods miss. Emerging strategies like the body rebound exemplify the potential of maximum-entropy RL when done right: the agent is not simply “noisy” – it is thoughtfully stochastic. Related Efforts and Broader Impact. FastDSAC is not the only recent attempt to make high-throughput humanoid learning faster or more stable. Another contemporaneous work (also called FastDSAC, by Lu et al., 2026) tackled a related problem: how to train off-policy RL under massive parallel sampling without destabilizing value estimates. Lu et al.’s approach introduces a truncated Gaussian policy that explicitly bans extremely large actions during learning. Their idea is that in a high-update setting, rare outlier actions (drawn from the tails of a Gaussian) can severely bias the targets; by clipping those out, they regularize the critic. Our focus is orthogonal: DEM reshuffles noise rather than trimming it. Both lines share the intuition that naive Gaussian proposals can be problematic at scale. FastDSAC (Xue et al.) shows that we can keep the full stochastic policy, but smarter; Lu et al. shows we can also simply eliminate the too-extreme corners. It would be interesting in future work to see if these ideas can be combined. Aside from that, FastDSAC builds on a lineage of maximum-entropy RL methods. Soft Actor-Critic itself was a major milestone for robust continuous control, but it was typically applied to lower-dimensional tasks (like half-cheetah or simple quadrupeds). Prior efforts in high-DoF control often defaulted to deterministic algorithms or special techniques. FastDSAC demonstrates that scale – high-throughput simulation and powerful networks – plus structure – DEM and distributional critics – can finally allow the original SAC paradigm to scale up. In some sense, this work returns to SAC’s roots in robotics while modernizing it for the new era of massive compute. One could also see FastDSAC as part of a broader trend toward adaptive exploration in RL. Other recent papers (e.g. using diffusion models or adaptive noise heuristics) recognize that fixed Gaussian noise is too blunt for complex tasks. FastDSAC’s contribution is to learn the adaptation itself as part of the policy. We anticipate that the idea of per-dimension entropy control will inspire follow-up work: for example, similar schemes could benefit manipulation tasks, quadrupeds, or even multi-agent domains where some agents or limbs should explore more than others. Conclusion. FastDSAC shines a light on an exciting possibility: that maximum-entropy policies can succeed in very high-dimensional robotics tasks if the entropy is managed cleverly. By decomposing the exploration noise and learning to focus it dynamically, and by using a richer value function, FastDSAC turns previously wasted randomness into a powerful exploratory force. The results show that a carefully designed stochastic policy is not only viable but often superior to deterministic ones in complex humanoid domains. Indeed, FastDSAC’s success invites us to rethink standard wisdom: rather than avoiding “noise,” we should embrace it – but in the right places and to the right degree. For practitioners, this work suggests a new recipe for training humanoids: use SAC-like objectives, but augment your policy with an entropy allocation mechanism and a distributional critic. The code (or future releases) will hopefully let roboticists plug FastDSAC into their own simulators. Perhaps even more broadly, it signals that maximum-entropy RL – long touted for its robustness and exploration – has been held back only by engineering barriers. With methods like FastDSAC, those barriers may finally be falling, unlocking a new horizon of diverse and stable learned behaviors on real robots. References: The ideas in this article are drawn from the FastDSAC preprint by Xue et al., 2026 (arXiv:2603.12612), including key details on DEM and the continuous critic. Performance figures are as reported (e.g. ~180% and 400% improvements on Basketball and Balance tasks). Related methods (Lu et al.’s FastDSAC) are also noted for contrast. The underlying Soft Actor-Critic framework we build on is detailed in Haarnoja et al. (2018).