Latest / Embodied AI 101 / HY-Embodied-VLM-1.0: Efficient Physical-World Agents

Transcript

HY-Embodied-VLM-1.0: Efficient Physical-World Agents · Embodied AI 101

• Hy-Embodied-VLM-1.0: Efficient Physical-World Agents. Physical-world robots face very different challenges than purely digital AI. A domestic or industrial robot needs to understand 3D spaces and the physics of objects, reason about how its own actions will change the environment, and adapt on the fly as things inevitably go wrong. In contrast, traditional vision-language models (VLMs) have been mainly trained on static images and text and excel at broad perception or text reasoning tasks, but often lack the “action-centric” understanding needed for operating in the real world. The new Hy-Embodied-VLM-1.0 (by Tencent Robotics X and Hunyuan) is explicitly built to bridge that gap. It is a vision-language-action foundation model designed for robots and embodied agents, trained to “see” physical scenes, plan actions, and even explain or correct its behavior in real time. According to Tencent’s report, Hy-Embodied-VLM-1.0 delivers strong results on 38 diagnostic benchmarks plus real-world navigation tests, all while running with a very lean footprint – only about 3 billion active parameters – thanks to a Mixture-of-Experts design. In this episode we’ll unpack what Hy-Embodied-VLM-1.0 actually is and how it was built. We’ll cover the key innovations in its architecture (a sparsely-activated MoE language backbone and a high-resolution vision encoder), the multi-stage training regimen (combining large-scale pre-training, chain-of-thought fine-tuning, and specialized RL), and the novel “action-centric” capability taxonomy that guided its development. Along the way we’ll look at its benchmark performance (what tasks it excels at) and how it behaves in closed-loop navigation tests. Finally we’ll discuss why this model matters: it shows a path toward high-level embodied reasoning without enormous compute budgets, opening the door to next-generation robots that see, plan, and adapt in the real world. Why an Embodied Foundation Model?. Most large vision-language models today (like GPT-style agents, open-vocabulary detectors, or multimodal assistants) have been optimized for tasks like image understanding, question answering, or on-screen gameplay. But controlling a physical robot is more demanding. As the Tencent team points out, robotic agents must “understand three-dimensional spatial structures, reason about the consequences of physical actions, and adapt to dynamic changes in real-time”. A conversational VLM that answers questions about images only needs to describe what it sees, whereas a robot might need to know where exactly an object is, how it can grasp it, what happens if you move it in some way, and what sequence of moves will achieve a goal. For example, say you ask “Can you pick up that mug and put it on the shelf?” A digital agent might struggle to ground the question in a 3D scene. An embodied agent, by contrast, needs an internal notion of the mug’s grip points (affordances), the spatial layout, the robot’s own capabilities, and the future consequences of its actions. It also needs to reason about sequences of actions (e.g. grasp mug → lift → move → release) and handle uncertainty (e.g. if the mug slips, figure out an alternative). Building one monolithic model that can handle all of that – vision, language, planning, reasoning – is an open challenge in embodied AI. Hy-Embodied-VLM-1.0 is Tencent’s entry into this space. It’s part of a new wave of embodied foundation models – the idea being: just as GPT and other large language models gave us a powerful general “brain” for text tasks, an embodied VLM is a general-purpose “brain” for physical agents. Tencent calls it an embodied foundation model, specifically tailored for real-world robots and agents. In practice, Hy-Embodied-VLM-1.0 takes as input both visual data (images from the robot’s cameras) and text (instructions or queries) and outputs text (like an answer or a plan) or action commands in a structured way. The model is essentially a vision-language model (with a large vision encoder and a large language model backbone) but trained and evaluated on robotics tasks, with special attention to efficient operation on a robot. Tencent reports that it achieves competitive results across 38 diagnostic benchmarks, and even in closed-loop navigation tasks, while keeping only about 3 billion parameters active during inference. That efficiency comes from a Mixture-of-Experts (MoE) architecture (we’ll explain that below) which allows the model to have a much larger capacity (≈30B parameters total) but only “wake up” a small subset on any given input. In short, this is a model built to answer the question: How can we give a robot a VLM-like brain that can actually see and act in the real world? The next sections break down the answer. An “Action-Centric” Capability Taxonomy. To guide the design and evaluation of their agent, the authors start with an explicit capability taxonomy – a multi-level categorization of the skills a physical agent needs. This is one of the novel ideas in their approach. Rather than treating everything as just more VLM fine-tuning data, they structured the training around three progressive tiers of capabilities (Figure 2 in their report):. Action-Relevant State Understanding: The basic perception layer. The agent must identify objects and their properties as they relate to action. This means understanding not just that “there is a mug on the table,” but recognizing parts (the mug’s handle, the rim, the inside), infer depth and spatial relations (how far behind or off-center it is), and crucially, identify affordances (e.g. which areas can be grasped or which surface is flat enough for stable placement). In other words, the model must focus on features that affect how it can interact with the scene – much more fine-grained and actionable than just generic image captioning. As the paper’s taxonomy explains, this tier covers things like: identifying object parts and attributes, reasoning about 3D geometry and distances, and understanding affordances (graspable regions, functional parts). Action-Transition Reasoning: Once the agent knows the state, it needs to reason about the dynamics – how will things change if I act? This means moving beyond static description to predictive understanding. Key skills here include: Interaction understanding (e.g. recognizing hand-object or human-robot interactions and social cues like pointing or gaze), action decision and grounding (choosing the next action in a task and mapping it to precise coordinates or trajectories), and physical feasibility/prediction (assessing whether an action is possible and predicting the immediate outcome). For example, before reaching to pick up the mug, the agent should reason, “is the mug stable on the table? Will my arm collide with anything? If I rotate the wrist, will the mug tilt or spill?”. In this tier, the model is explicitly trained to think about postconditions and action outcomes, not just object labels. Sequential and Adaptive Reasoning: The highest tier covers long-horizon execution and planning. Real-world goals often require many steps (e.g. “prepare a meal” → chop vegetables → cook → serve) and conditional behavior. The model must be able to compose actions into a plan and also to recover when things go wrong. Concretely, this includes skills like long-horizon decomposition (breaking a goal into sub-tasks), Vision-Language Navigation (VLN) (procedurally following spoken/written instructions through an environment), and reflection & repair (detecting failures, analyzing causes, and replanning). For instance, if the agent tries to grasp the mug and it drops, it should diagnose “I slipped” and plan an alternative. These three tiers – state perception, action reasoning, sequential planning – form a progressive hierarchy of competence. By framing experiments and data collection around these categories, the researchers can systematically test “does the model understand object poses?” “does it reason about the result of a push?” “can it plan a multi-step route?” In fact, their diagnostic benchmarks are grouped along these lines. We’ll see below that Hy-Embodied-VLM-1.0 excels in all three categories, reflecting this taxonomy-driven design. Model Architecture: Mixture-of-Experts and Hy-ViT2. With the capability taxonomy established, what model architecture did they use? Hy-Embodied-VLM-1.0 is built on two main components: (a) a large vision encoder (called Hy-ViT2) and (b) a large language model backbone (called Hy3-A3B) augmented with a Mixture-of-Experts (MoE) design. The goal is to give the model very high capacity for reasoning, but with much less compute at inference. Vision encoder (Hy-ViT2): The model uses a Vision Transformer (ViT) that operates at native resolution. In practice, that means it can take images (or video frames) of varying size and aspect ratio and process them without excessive resizing or cropping. This is important for robotics because fine visual details (the small lever on a panel, or the exact edge of a knife) may matter. Hy-ViT2 is a distilled model with ~400 million parameters, trained to be efficient on edge devices but still accurate. It was distilled from an even larger ViT to preserve precision. Notably, the authors include a “visual latent token” approach (inherited from the HY-Embodied-0.5 model) to bridge vision and language: each image gets an extra learned token that summarizes global features for the language part to attend to. The bottom line: Hy-ViT2 ensures that visual perception – object details, depth cues, fine textures – is captured at high fidelity, directly feeding into the embodied reasoning. Language backbone (Hy3-A3B) with MoE: The core of Hy-Embodied-VLM-1.0 is its transformers-based language model. Tencent calls it Hy3-A3B, which means (approximately) “Hunyuan 3rd generation, active 3 billion parameters.” Internally it actually has about 30 billion total parameters – comparable to a large LLM – but only about 3 billion of them are activated on any given token. That is the power of the Mixture-of-Experts (MoE) architecture. In a standard dense model, every parameter contributes each time. In an MoE design, the model contains many “expert” sub-networks, but a learned gating mechanism selects only a few of them per input. In Hy-Embodied, this means that although 30B parameters exist, each input token is routed through only ~3B worth. The result is that during inference the model’s speed and memory cost are closer to a 3B model, even though the underlying capacity is far larger. In practice Tencent reports that this MoE trick allows Hy-Embodied-VLM-1.0 to “retain the extensive knowledge and reasoning depth of a 30B-scale model while operating at the speed and cost of a 3B-scale model”. This parameter sparsity is crucial for real robots, where compute and latency are limited, but we still want deep reasoning chains. Why is MoE especially useful here? The designers specifically note that latent, insight-like reasoning (the kind needed for planning and prediction) is sparse by nature – not every piece of reasoning is needed for every problem. By gating experts on relevant parts of the task (e.g. certain experts specialized in spatial reasoning, others in counting, etc.), the model becomes both high-capacity and efficient. This is somewhat analogous to Switch Transformers from Google, or the mixture-of-transformers (MoT) idea from the 0.5 model, but here implemented at the language level. The net effect is that Hy-Embodied-VLM-1.0 can handle very complex chains of thought (like multi-step planning) without requiring a massive dense compute. Putting it together, the final model pipeline is: Image (processed by Hy-ViT2) → Vision tokens + visual latent token → fusion with text input tokens → Hy3-A3B MoE language model with CoT decoding. It’s still an encoder-decoder model in principle (it generates text or structured responses), but the emphasis is on preserving spatial detail and routing through a huge pool of experts. The result is what the team calls an embodied VLM, capable of both perception and action reasoning. Training Regimen: Self-Evolution Through Pretraining, SFT, and RL. Creating such a model requires more than just picking an architecture; it needs careful training on the right data. The Hy-Embodied team describes a self-evolving, multi-stage training loop (Figure 3 in their paper) to instill embodied reasoning step by step. These stages are:. Pre-training and Supervised Fine-Tuning (SFT): First, the model is given “general embodied knowledge” through large-scale pre-training on multimodal corpora. This likely includes billions of image-text pairs and many robotics-themed examples, so that the model builds a baseline 3D and common-sense understanding. Then comes supervised fine-tuning on curated datasets. Crucially, in SFT they incorporate chain-of-thought (CoT) traces: instead of just feeding an input (e.g. an image and question) and training for the correct answer, they provide example reasoning steps. The model is taught to articulate its reasoning (the “thinking” chain) before giving the final answer. This mimics how we train language models with CoT prompts – it encourages the model to develop internal reasoning processes. The supervised data is split to emphasize the taxonomy: some instances focus on spatial perception (“what are these objects?”), others on short causal reasoning (“what happens if A is pushed?”), and so on, aligning with each capability tier. Reasoning-Elicitation Reinforcement Learning: After SFT, the model’s reasoning is further refined via reinforcement learning. They use a custom algorithm called Group-Relative Policy Optimization (GRPO). In each RL round, the model generates multiple possible responses or action sequences for a given prompt. A reward function scores each by how well it satisfies the task (e.g. accuracy of predicted trajectory, completeness of navigation). Then GRPO normalizes these rewards within each group of sampled responses, effectively letting the model compare each candidate against others in the same batch. This group-relative advantage formulation helps stabilize training and focus on the best completions. Concretely, if the model generates several possible “thinking traces” or plans for a scenario, GRPO encourages shifting probability mass toward the responses that yield correct and verifiable outcomes. It’s like saying “among the things you tried, prefer the ones that outright solved the task or made correct predictions.” The paper notes they reward things like correct object grounding, trajectory accuracy, and textual match with ground truth. After many updates, the model discerns stronger policy parameters (denoted (\theta_{rl})) that consistently produce better reasoning steps. Specialization and Fusion: The policy from stage 2 is then used to generate even more training data. Through rejection sampling, the model’s current policy produces a set of high-quality chain-of-thought answers (those that pass the reward threshold). These CoT outputs form a new fine data set ((D_{cot})). The model is then re-trained (“consolidated”) on this dataset, producing a refined policy (\theta_{cons}). Finally, they run one more RL step specialized on different reward types: one copy of the model ((\theta_{geometric})) is optimized for continuous-valued precision rewards (like exact distance or alignment), and another ((\theta_{discrete})) is optimized for discrete success/failure rewards (like hitting a goal). These two specialists are then merged (“fused”) into the final model (\theta_{final}) that powers Hy-Embodied-VLM-1.0. In effect, stages 2–3 perform an iterative self-play/data-generation loop with the model refining itself. The model discovers better reasoning patterns, then those patterns are distilled back into the model. This is reminiscent of OpenAI’s pipeline where a pretrained model generates data for its successor. Here it happens within a single training regimen, giving the agent progressively sharper embodied reasoning. In addition to the algorithmic steps above, the team also assembled massive training datasets aligned to the taxonomy. They report collecting on the order of tens to hundreds of millions of examples. For instance, in a dedicated “mid-train” stage they accumulated over 18 million question-answer pairs covering state understanding, action effects, and reasoning; then in a “post-train” stage about 48 thousand high-quality instruction-following examples focused on tasks like counting, spatial queries, navigation, and failure analysis. These staged datasets ensure the model sees plenty of both shallow perception tasks and deep reasoning tasks during training. Overall, the picture is that Hy-Embodied-VLM-1.0 is trained from the ground up as an embodied agent: first exposed to raw multimodal data, then carefully finetuned to reason step-by-step, and finally polished with reinforcement feedback. The result is a model whose internal “thought process” (guided by CoT supervision and RL) aligns well with its high-level objectives of perception, planning, and action. Evaluation: Benchmarks and Navigation. How well does all this work? The authors put Hy-Embodied-VLM-1.0 through a battery of tests, both offline benchmarks and simulated navigation tasks. Diagnostic Benchmarks: They assembled a suite of 38 tests that cover various aspects of embodied intelligence. These include tasks checking visual understanding (e.g. depth estimation from images, object recognition in clutter), state reasoning (e.g. will moving an object here collide with something else?), action outcome prediction, simple planning problems, and multi-step puzzles. Importantly, the benchmarks are grouped into the three taxonomy categories we discussed (state understanding, action-transition reasoning, sequential reasoning). The results are very strong, especially given the model’s efficiency. Hy-Embodied-VLM-1.0 “A3B variant” averaged an overall score of 65.6 across all tasks, beating comparable models like Qwen3.6-A3B (61.2) and its own earlier-generation cousin Hy-Embodied-0.5 (57.2). In fact it was the top performer in each of the three capability categories:. State Understanding: Score 68.6 (highest rank on 11 out of 23 state tasks). Action-Transition Reasoning: Score 64.1 (top on 5/8 tasks). Sequential Reasoning: Score 57.4 (top on 3/7 tasks). These numbers tell a promising story: the model not only is generally stronger, but it excels at basic perception and causality tasks (the first two categories) and holds its own on longer-horizon planning. For example, one diagnostic involved reading Roman numerals on a clock, and Hy-Embodied-VLM-1.0 did so correctly more often than any baseline. Another asked itself “which of these two objects is closer to the camera?” in a cluttered scene – again it topped the charts. Even qualitative failure analysis is tested: in one example an image showed a robot arm that had tried and failed to insert a cylinder into a hole. The model could correctly diagnose the reason for failure (e.g. misalignment or interference). It’s not just pointing at objects, but explaining actions in context. In short, against established VLM-based baselines (including recent high-end models), Hy-Embodied-VLM-1.0 sets new state-of-the-art marks on most calibration tasks. The reported average gain is +4.4 percentage points over the strongest comparable baseline, and +8.4 points over the previous HY-Embodied-0.5 model on the same tests. That’s substantial, especially since Hy-Embodied-1.0 does this while only activating 3B parameters versus 32B in the old model – a roughly tenfold efficiency improvement for similar capacity. Closed-loop Navigation: The team went further by deploying the model in the loop on simulated robotics tasks. They plugged it into a standard vision-language navigation (VLN) setup: a virtual 3D house (Matterport3D) and natural-language path instructions or goal specifications. Here the model must repeatedly parse an instruction or goal and decide where to move in the environment, loop by loop. In these tests Hy-Embodied-VLM-1.0 again performed well. On the Room-to-Room instruction-following benchmark (R2R-CE setup), it achieved a 57.9% success rate. In practical terms, this means that more than half the time it could follow a complex trajectory (“pass the mirror on your right, then go into the hallway.”) from start to goal in one try. For context, pure LLM-based navigation agents often score much lower without environmental feedback. The Tencent report emphasizes its “high path efficiency and reliable goal verification.”. They also tested zero-shot object-goal navigation: given a target category (like “find a cushion somewhere in this unseen home”), the model guided the agent through the house to a likely cushion location. It succeeded 38.3% of the time, which is impressive for a zero-shot VLM agent and is said to beat larger baselines. In navigation examples (Figure 4 of their report), the agent is shown making step-by-step decisions (“turn left at the couch, go down the hall, entering the kitchen”) and using visual cues (e.g. “I see a coffee table, I should go past it to find a cushion”). This indicates that the model’s internal memory of past views plus its spatial grounding really kicks in. Together, the navigation results show that Hy-Embodied-VLM-1.0 isn’t just good on paper: it works as a control policy too. It can parse high-level instructions, ground them into world coordinates, and react to new observations. And crucially, it does so on the fly, demonstrating fast inference and low latency. Significance: A New Blueprint for Efficient Agents. Why does all this matter for the broader field of embodied AI? Hy-Embodied-VLM-1.0 is notable for a few reasons:. First, it’s very efficient for its power. Embodied AI practitioners often face a trade-off: either run huge models in the cloud (with latency problems) or use small models on edge devices (sacrificing capability). By using MoE and this structured training pipeline, Tencent shows that you can have your cake and eat it too: near top-tier performance with only 3B active parameters. As the authors put it, “high-level intelligence does not always require massive computational overhead”. In other words, smart design can compress an “effective model size” of 30B into the speed of 3B. This is a big deal for real-world robotics, where carrying around a rack of servers isn’t an option. It suggests we may see more practical embodied AI systems that can run on-board with reasonable hardware. Second, the work emphasizes an action-centric perspective throughout. Rather than adding robotic tasks to a century of generic vision-language data, they explicitly categorized and trained for the transitions and plans that matter to agents. This methodological clarity is valuable. The taxonomy and staged dataset preparation mean that future researchers have a blueprint: define what your robot needs to do, collect data in those modalities, and train iteratively. In their words, Hy-Embodied-VLM-1.0 provides “a blueprint for efficient agents that can understand, act, and recover in the complex, unpredictable physical world”. This contrasts with earlier VLM-only approaches that stopped at perception; here perception, planning, and adaptability are integrated. Third, it opens doors for end-to-end robotic systems. As one of the earliest open-source embodied VLMs, it can be plugged into ongoing research on vision-language-action (VLA) pipelines. For example, practitioners can now use Hy-Embodied-VLM-1.0 as the “brain” in robot frameworks, providing it with scene images and receiving action plans or diagnostics. The Tencent team has already released code and weights on GitHub and HuggingFace, making it readily accessible. There’s also a companion model, Hy-Embodied-RxBrain-1.0, oriented toward imagining future states, suggesting a larger ecosystem of embodied AI tools now becoming available. Finally, this work nudges the community toward a long-envisioned goal: unify perception, language, and action in one model. We’ve seen smaller steps (DeepMind’s Gato, for instance, was a multi-task agent but on toy tasks), and concurrent efforts (like the other Tencent Hy-Embodied 0.5, or the new Hydra or V-LM models), but Hy-Embodied-VLM-1.0 is a landmark in scale and ambition. It shows that large-scale LLM techniques can be adapted for “physical reasoning” at high speed. The improvements over both past Tencent models and external baselines (e.g. Qwen, Cosmos) indicate there are real gains from thinking about embodiment at every stage. In summary, Hy-Embodied-VLM-1.0 marks a significant step toward truly generalist robot models. It demonstrates strong embodied reasoning – from fine-grained vision to long-horizon planning – while being deployable. For robotics researchers, it’s a new tool and proof-of-concept that the embodied foundation model approach is viable. One might expect future iterations to add actual motor outputs or tighter sensorimotor loops, but as it stands Hy-Embodied-VLM-1.0 is already a highly capable “brain” for agents, and its open release will no doubt spur further innovation. Sources: The above account is based on the Hy-Embodied-VLM-1.0 technical report (Tencent Robotics X, 2026) and press coverage. Key results and design details were drawn from Tencent’s published paper and supporting materials. The model and code are publicly available for experimentation.