Latest / Embodied AI 101 / Does VLA Even Know the Basics? Act2Answer Benchmark

Transcript

Does VLA Even Know the Basics? Act2Answer Benchmark · Embodied AI 101

• Does VLA Even Know the Basics? Act2Answer Benchmark. Vision–language models (VLMs) have become incredibly powerful at tasks like image captioning, visual question answering, and general scene understanding. In parallel, the robotics community has begun extending these capabilities into the physical world by training Vision–Language–Action (VLA) models. A VLA is essentially a multimodal “foundation” model that takes an image of the robot’s surroundings and a text instruction as input and directly outputs robot actions. In practice VLAs are built by taking a pretrained vision-language model and fine-tuning it on robot task data (often via imitation or reinforcement learning) so that it learns to turn phrases into movement trajectories. In other words, the same model that once answered questions about images is now being trained to pick up and place objects or move through an environment. But this raises a critical question: Does a VLA still “know” the same world knowledge after this fine-tuning? Ideally we’d like our robots to retain all the broad vision-and-language understanding of their backbone VLMs, even after they’ve learned new control skills. It would be disappointing if a model could assemble furniture or sort objects, but forgot basic facts and commonsense in the process. Until now this issue had barely been studied: success on robotics benchmarks often only checks if a robot can do a manipulation task, without ever asking “Does it remember facts about the world?” The new Act2Answer benchmark directly addresses this gap by testing VLA models with a suite of “question-answering via action” tasks. In short, Act2Answer asks VLA agents to answer knowledge questions by placing an object on the correct answer image, turning language queries into a simple robot action. By doing so, it measures how much of the original VLM’s knowledge the agent has retained after being fine-tuned for robotics. In this episode we’ll dive into the Act2Answer protocol and its findings. We’ll see that, unfortunately, many VLAs fall behind their VLM ancestors by a wide margin on commonsense questions. In fact, the authors of the new paper find “substantial gaps (≈20–40 points)” on the rich semantic question categories when comparing a VLA to its source VLM. VLAs still ace the really straightforward perceptual questions (e.g. “which block is red?” or “which shape is round?”), but they struggle a lot on deeper knowledge categories (e.g. living-world or cultural questions) that their original VLMs handled easily. We’ll also learn that co-training on visual question-answer data helps maintain this knowledge, and that the agent’s latent “belief” about the answer tends to reside in the middle layers of the network, fading out by the final action layers. By the end, we’ll have a clear picture of what Act2Answer reveals about current robotic VLAs: namely, that naive fine-tuning alone can hollow out their world knowledge, and that smarter training strategies will be needed if we want physically capable robots that still know the basics of the world. VLA Models and the Problem of Forgetting. Before we describe the benchmark, let’s set the stage. A Vision–Language–Action (VLA) model is essentially a robot controller built on top of a vision-language foundation. Formally, a VLA takes as input an image (or video frame) of the robot’s environment plus a text command or question, and outputs low-level control actions. For example, a VLA might see a cluttered workbench and get the instruction “put the hammer next to the red block,” and it must translate that into motions of a robot arm. In practice these models are usually constructed by fine-tuning a pretrained Vision–Language Model (like an LLM with visual input) on a dataset of robot demonstrations or policies. In other words, the VLA retains the original model’s image-language encoder (the “brain” that understands pictures and words), and adds or modifies an output head that drives the actions. This approach has delivered some impressive robotics results. Fine-tuned VLAs can perform tasks like object manipulation and navigation with natural language commands. They seem to understand instructions and scenes in a flexible way. But it’s not obvious how deeply they still understand the scene, beyond what’s needed for that task. In machine learning terms, this setup is akin to catastrophic forgetting: you take a model with lots of learned knowledge and then train it on a new task, and inevitably some of the old knowledge may be overwritten by the new learning. Crucially, standard robotics benchmarks don’t measure this knowledge retention. They check if the robot successfully stacks blocks or follows a navigation path. If the robot fails a knowledge-heavy query (say, “which object is normally used to cut paper?”), is it because it forgot what scissors are, or just because it can’t carry out the required movement? These two failure modes look similar in a robotics trial, but they have very different implications. If a robot truly doesn’t know that scissors cut paper, it needs a better understanding of the world – not a better arm controller. To disentangle the two, we need a test that isolates the knowledge component from the control component as much as possible. That is exactly the idea behind Act2Answer. The protocol transforms language questions into one-step physical actions, so that answering the question doesn’t involve a complex plan, only the simplest possible action. This lets us focus on the model’s knowledge and reasoning, rather than its control proficiency. The Act2Answer Protocol. The key idea of Act2Answer is simple: turn a multiple-choice question about the world into a very short tabletop episode, where the agent shows its answer by physically placing an object. In each episode, the robot (simulated) sees a question instruction and also sees the candidate answers as images or symbols placed at distinct, known locations on a table. Then the robot must pick up one end of a block or cube and place it on the location corresponding to its chosen answer. If it picks the correct answer image, the episode is scored as a success; otherwise it’s a failure. Because each episode is just a single object-placement action, the “control” part of the task is trivial: the main challenge is picking which answer is right. For example, imagine an Act2Answer episode for a color question. The scene might have three colored plates or squares on the table – red, blue, and green – each with an image of the color or simply colored itself. The instruction says something like: “Move the cube onto the [RED] plate.” (If it is a question like “Which color is this sky?”, the plates could be labeled or pictured, and the robot chooses by placing the cube on one of them.) The only thing the robot must do is push the cube onto one of these plates. If it goes to red when red is indeed correct, that’s a success. No tricky multi-step movement or precision placement is needed – it’s effectively a one-step classification by action. By construction, Act2Answer minimizes control confounds. The robot doesn’t have to navigate far, avoid obstacles, or perform any complicated dynamics; it just needed to place one object. This makes the measure an action-grounded knowledge metric. If the agent fails, it’s likely because it “thought” the wrong answer image was correct, not because it can’t walk or grasp. Formally, each Act2Answer episode is drawn from a standard vision-language benchmark question. The authors take a VLM-style question (e.g. “Which of these objects is used for cutting paper?” with options “scissors” vs “glue gun” vs “hammer”), and create a virtual scene by placing images of each answer option at designated spots in front of the robot. In the center is a distinct “answer plate” for each choice. The robot’s instruction is simply the original question phrased as a language command (like a question or imperative). The agent then must control its arm to move a cube onto the plate corresponding to what it believes is the right answer. If the cube ends up on the correct plate, that trial is counted as correct. Otherwise, it’s an error. The overall metric across episodes is the action-grounded success rate (percentage of episodes answered correctly). Importantly, Act2Answer covers many types of knowledge. The authors assemble questions from a wide variety of VLM benchmarks: basic perception tasks (“Which object is blue? Which shape is round?”), commonsense and world knowledge (“Which item is used to open a bottle?”; “Who is a real celebrity vs. a cartoon?”), temporal or cultural facts (“Which number is larger, 12 or 21?”; “What day comes after Tuesday?”), and so on. These are organized into categories like color, shape, counting, emotion, living world, traffic rules, public info, celebrity, etc. In total they built over 1,700 episodes spanning 12 knowledge categories. Each category has dozens of questions turned into simple pick-the-answer actions. The hypothesis is that a well-retained VLM backbone should still ace these questions via action, while a model that lost its knowledge will perform poorly. Figure 1 of the paper (not shown here) illustrates some example episodes. In one “color” episode, the agent might see a “Which block is blue?” instruction and two colored plates. In an “attribute” episode, it might have plates labeled by objects (“Which is a vehicle?” with pictures of bicycle vs. apple). In an “emotion” episode, plates could show a smiling vs frowning face with the question “Which face is happy?” In a “celebrity” episode, plates show a photo of a real person and a cartoon character with “Which one is a real person?”. Each time the agent must move the cube to one plate. Because the environment is so simple (one block, one push action), the agent’s answer is clear evidence of its underlying knowledge or reasoning. The authors describe Act2Answer as a “lightweight protocol” that adapts existing VLM knowledge tests to the embodied setting. It requires no new language understanding from the model – just the ability to decide which option is correct – but it does require the model to output an action instead of a text answer. This clever twist ensures that the evaluation is action-based (hence the name “Act2Answer”), yet still really measuring the agent’s language-based knowledge. Test Suite and Models Evaluated. Using Act2Answer, the authors curate a comprehensive benchmark. As mentioned, it covers 12 categories. These include:. Basic Attributes: Colors, shapes, symmetry, etc. (e.g. “Which object is red?”, “Which pattern is symmetrical?”). Counting: Simple number tasks (e.g. “Are there more red blocks or blue blocks?”). Temporal/Logical: Time-related and logic (e.g. “Which is earlier, Monday or Friday?”, “Which number is even?”). Public Info / Cultural: General facts (e.g. “What is the emergency number?”, “Which person is a UK prime minister?”). Traffic/Signs: (e.g. “Which sign means stop?”). Celebrity vs Fiction: Distinguishing real people from cartoons. Emotion: Identifying emotional expressions. Living World: Biology/animal knowledge (e.g. “Which is a mammal?”). Norms: Social or practical norms (e.g. “Which object is a tool used for cutting?”). These categories capture both perceptual and deeper semantic knowledge. They come from standard VLM QA datasets or knowledge trivia pools. Each question is turned into an episode with two or three answer choices. In total there are roughly 168–200 episodes per category (about 1,720 overall). This provides a broad testbed: a truly grounded agent should be able to answer any of these. Against this suite, the authors evaluate a range of models. They include seven state-of-the-art VLA models (the exactly tested models come from contemporary research and industry): for example, π₀, OpenVLA, Magma, Xiaomi-Robotics-R0, InternVLA-M1, SmolVLA, and SpatialVLA. Each of these started from a powerful VLM backbone (like Qwen-VL, InternLM-VL, etc.) and was fine-tuned for robotics in different ways. Alongside, they test nine pure VLM baselines (the underlying vision-language models without the robotics fine-tuning). These VLMs serve as the ideal “source of knowledge.” To evaluate the VLMs on the same task, the authors effectively let them pick an answer directly (for example by prompting the model to choose or by seeing which answer is most likely) – because the VLMs cannot move a robot arm, they are scored by checking if they would pick the correct answer text. So one can compare each VLA’s accuracy to what its base VLM “would have gotten.” In ideal world, a VLA would match its VLM’s performance on these questions (since it should still know the answer), but that is exactly what the study is testing. Major Findings: Gaps in Knowledge Retention. The headline result is that vision-language-action fine-tuning tends to erode world knowledge compared to the original VLM. Across the board, all the tested VLAs lag behind their pure VLM counterparts on the Act2Answer suite. Quantitatively, the gaps are large: on the order of tens of percentage points. For example, simple categories like color or shape remain nearly perfect (VLAs might score in the 80–90% range matching VLMs), but on richer semantic categories like living world, temporal, celebrity, norms, etc., the VLA scores can drop by about 20–40 points compared to its VLM. In one stark quote from the conclusion, the authors note “a large performance gap exists between source VLMs and their VLA counterparts, confirming knowledge attenuation during robotics fine-tuning”. Put another way: If a VLM baseline on a category is, say, 90% correct, the corresponding VLA might only be 50–70% on the same questions – despite having been built from that VLM. This is not a small quirk; it’s systematic. The more sophisticated the knowledge required, the wider the gap. The VLAs remain solid on the trivial perceptual stuff (color and shape questions), but even modest reasoning tasks see notable declines. For instance, questions in the Living World category (distinguishing animals, plants, etc.) or Public Info (knowledge of common facts) show some of the biggest drops. In contrast, the pure VLMs (with no robotics fine-tuning) breeze through those same questions at a very high success rate. The authors systematically break down performance by category. One key finding is that “VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs”. In plain English: if it’s a low-level visual thing they can handle it, but anything needing world knowledge or common sense is hit. The trend is consistent across all the VLAs they tried. This result highlights a kind of trade-off: focusing on low-level motor adaptation (the actual manipulation tasks) seems to come at the expense of higher-level knowledge. Importantly, the evaluation protocol (Act2Answer) makes sure this isn’t just about a specific architecture or data. The pattern held across multiple model families. It suggests a general phenomenon: “fine-tuning VLMs on action data is insufficient to preserve all their original knowledge,” as the authors conclude. One can think of this intuitively: the loss function during VLA training cares about executing tasks correctly, not about answering factoid questions. So the model parameters will drift toward what’s important for control, and the pressure to remember unrelated facts vanishes. Without an explicit requirement to keep the original Q&A ability, the fine-tuning process all but guarantees forgetting at least some of that content knowledge. The results confirm this suspicion quantitatively. A Concrete Example: Answering by Action. To ground this, imagine a specific example from Act2Answer. Suppose we have a “Celebrity vs. Cartoon” episode. The question might be “Which image is a photograph of a real person?” and the two plates show (1) a photo of a well-known actress, and (2) a cartoon drawing of a similar-looking person. A bona fide VLM (like GPT-Vision) would very likely answer “the photo of the real person.” But after being fine-tuned into a VLA on assorted robot tasks, that same model might hesitantly choose the cartoon instead, simply because the association between the visual features and the concept “real person” was not enforced during robot training. In Act2Answer, that mistake would correspond to the agent placing its cube on the cartoon plate instead of the real person’s plate – a clear failure of knowledge, not control. Contrast that with a “Color” episode. The question might be “Which ball is blue?” and the plates display a blue ball and a red ball. Even a heavily fine-tuned VLA presumably will succeed here, because “blue” is a visual word grounded in the image, and the robot only needed to match color. Indeed, the benchmark found nearly all VLAs matching their VLMs on these visual categories. It’s the more conceptual ones – not immediate pixel-level tasks – where things fall apart. Saving Knowledge with Co-Training. Interestingly, the study uncovers a mitigating factor: training supervision matters. The authors note that the VLAs which had been taught some visual question-answering (VQA) tasks during fine-tuning retained much more knowledge. In other words, if your VLA is not only trained to move blocks but also occasionally asked to answer a question (perhaps the same Act2Answer questions!), it won’t forget as much. One way to see this is that some models in the study were trained with an auxiliary VQA objective. They still got fine-tuned on robot data, but also continued to answer image-based questions (maybe from a dataset like GQA, VQAv2, or similar) at the same time. Those multi-task VLAs performed significantly better on the Act2Answer benchmark than identical models trained only on robot data. The authors explicitly mention that “including continued vision-language supervision (VQA co-training) is associated with better knowledge-sensitive performance”. This makes sense: by keeping the model’s brain “exercised” on language–vision problems, we force it to preserve those capabilities. It echoes a known trick in transfer learning: if you fine-tune too heavily on a narrow dataset, you can catastrophically forget, but if you keep it multi-task or occasionally “remind” it of the original tasks, the forgetting is mitigated. In robotics, this suggests that simply training on control tasks isn’t enough; we should interleave or mix in other tasks (like Q&A) to keep the internal knowledge alive. Peeking Inside with Layerwise Probes. Another fascinating part of the paper is their layerwise analysis of the VLA networks. The authors perform “intent probing” at each layer to see where the answer is actually represented. Concretely, at each hidden layer of the vision-language encoder, they attach a small linear classifier (a probe) that tries to predict which answer the agent will choose. By measuring how well each probe does, they map where in the network the knowledge of the answer lives. What they find is telling: in the original VLM backbone, the information needed to answer a question is present fairly strongly in the mid-layers. But as you move up into the layers that have been specialized for action (the top or policy layers), that signal gets weaker and weaker. By the final action prediction layer, the direct trace of the VLM’s semantic understanding has largely faded. In their words, answer-relevant signals “peak in middle VLA layers but attenuate in upper layers”. This suggests a kind of bottleneck: the semantic meaning is parsed in early layers, but then the later layers discard much of it when focusing on controlling the arm. The action head apparently doesn’t fully carry along the rich semantic representation needed for general questions. The knowledge is “latent” in the mid-level features, but the final decision is guided by something else (maybe task-specific action affordances). Practically, this means that if one were to extract the answer directly from a VLA by looking at its top action output, one wouldn’t find much. The agent has to use the hidden layers to figure out the answer. The linear probes showed clues: e.g. in one model, layer 10 might still carry a strong signal about “the answer is plate #2” for a given question, but layer 15 (closer to output) does not. This confirms that fine-tuning pushes the knowledge aside when generating actions. The takeaway is that knowledge may not be fully “lost” in weight space, but it becomes inaccessible to the part of the model producing behavior. It’s similar to dumping data to a dimension the control head doesn’t bother reading. This reinforces the idea that pure end-task fine-tuning is breaking the alignment between “understanding” and “doing.”. Implications and Next Steps. Altogether, Act2Answer paints a somewhat cautionary picture of current VLA practices. On one hand, it’s impressive that these models can still answer simple perceptual questions through action. They clearly retain some ability to parse instructions and images. On the other hand, the glaring drop in knowledge on almost everything else shows that the standard pipeline – “take a VLM, fine-tune on robot data” – is not enough if we care about general knowledge. This has real consequences. Consider a household robot built with this technology: if it never hears trivia questions during training, it might vacuum your floor well but get stumped by which utensil to use for stirring coffee, or which component is safe to pick up. Those seem like basic capabilities if the robot is going to be versatile. The authors argue – and we agree – that future embodied AI needs training strategies that explicitly align semantic knowledge with action policies, rather than letting the latter wash out the former. What might those strategies be? The fact that VQA co-training helps is one clue: multi-task training that keeps asking the model questions could be baked into curriculum. Another approach could be architectural: preserving a separate fact-memory branch that isn’t overwritten. Some form of continual learning or replay might also prevent forgetting (e.g., occasionally retraining on old data). The authors even suggest that new objectives could be crafted to maintain knowledge – for example, adding an auxiliary loss that penalizes drifting away from the original VLM outputs on known questions. Any of these could be avenues to maintain knowledge in future VLA models. For now, Act2Answer itself is a valuable tool. The authors have released the suite and code (see the Act2Answer website) so that other researchers can evaluate their own models. Having such a benchmark means that any new VLA can be checked for basic knowledge retention as a standard practice, much like we check robots for success rates in manipulation tasks. Imagine after training your robot, you also run it on 100 trivia episodes to ensure it still “knows” the world. If it fails, maybe you adjust your training. In conclusion, “Does VLA know the basics?” The short answer is: not always. The paper’s experiments show that current VLAs often forget or underperform on commonsense and factual questions once they’ve been fine-tuned for action. That’s a wake-up call. It doesn’t mean VLM-to-VLA is a dead end, but it does mean that we have to be mindful of it. Vision-language backbones give us a lot of knowledge “for free,” so we should make an effort to keep it. Act2Answer provides the first concrete measure of this issue, and the results make it clear that more work is needed to bridge the gap between seeing, knowing, and doing in our embodied AI systems. Citations: The Act2Answer benchmark and results are described by Kachaev et al. (2026) in “Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models.” Key findings and methods are summarized above with reference to their abstract and analysis. Additional background on VLA models is drawn from the general literature.