Latest / Embodied AI 101 / AnoleVLA: Lightweight VLA with Deep State Space Models for Mobile Manipulation
Transcript
- AnoleVLA: Lightweight VLA with Deep State Space Models for Mobile Manipulation. Vision-language-action (VLA) models promise to let robots follow natural-language commands by directly mapping camera images and instructions into control trajectories. They’ve shown great potential: given an image and a command like “Place the apple in the red bowl,” a VLA can successively ground references (the “apple” and “red bowl”) and output motion commands to perform the task. But there’s a catch. Most high-performing VLAs today are built on huge transformer-based foundations (like LLaMA or PaLM variants) that have billions of parameters. Those models deliver accuracy, but they are bulky: they require lots of GPU memory and slow transformer attention is quadratic in sequence length. In practice, this means long inference times and high latency – a big problem for a mobile robot that needs to act in real time on limited hardware. Put simply, current state-of-the-art VLAs were trained to be smart, not necessarily to run fast on a robot. For example, one recent VLA called π₀.₅ uses a 3-billion-parameter language model together with discrete action tokenization and a separate action “expert” module. Another – an open-source VLA built on Llama-2 with 7B parameters – likewise looks impressive on benchmarks but is impractical for real robot deployment. Even so-called “compact” variants like the above π₀.₅ still carry billions of weights, and their transformer attention still costs 👀O(L²) for a token sequence of length L. In vision-language-action, the input is a concatenation of vision tokens, language tokens, and robot state tokens – and that can easily run into thousands of tokens to process at every timestep. All this adds up to a serious bottleneck for on-device robotics. The AnoleVLA paper by Takagi et al. directly addresses this challenge. The authors ask: Can we build a high-quality vision-language-action policy that is orders of magnitude cheaper to run? The key idea is graceful: replace the heavy transformer backbone with an efficient state-space model (SSM) architecture. State-space models maintain a small hidden state that gets updated incrementally as new tokens arrive, so processing happens in linear time O(L) instead of quadratic. Recent SSM variants (like the S4 model and its successors) have shown they can match transformers on long sequence tasks with much less compute overhead. AnoleVLA exploits this: it uses a “selective” deep SSM called Mamba as its core backbone. In effect, Mamba reads the vision, language, and proprioceptive tokens one by one, updating its state sequentially, rather than paying the cost of all-pairs attention. This change alone dramatically cuts latency and memory use. Despite its reduced size (only ~0.47B parameters), AnoleVLA manages to match or exceed the performance of much larger models. In both simulations and real robot tests, it achieved higher task success than a leading 3B-parameter VLA, and did so at roughly three times the inference speed. How? Beyond the backbone swap, AnoleVLA also predicts continuous robot actions directly (no discretization into tokens) and uses a two-stage training regime that improves control smoothness. Altogether, AnoleVLA shows that you don’t need to train or run a colossal transformer to do language-conditioned manipulation – a well-designed lightweight model can do it faster and sometimes even better. In the sections below we’ll walk through the problem setting, prior VLA approaches, the Mamba state-space backbone, and the architecture and training of AnoleVLA. We’ll also discuss its empirical results and why this approach matters for embodied AI on low-power robots. The Challenge of Language-Guided Manipulation. Imagine a home assistant robot: a user says “Can you put the apple in the red bowl?” and the robot must carry out a pick-and-place. This simple example actually involves a lot under the hood: the robot must visually detect objects, ground the referring expression “the apple” among clutter, plan a path to pick it, and then reach towards a designated container. Designing separate perception, planning, and control modules for every possible scenario is onerous. That’s why end-to-end VLAs – which combine vision, language and action in one neural network – have become popular. Pre-trained vision-language backbones (e.g. CLIP-style or image-instruct models) plus fine-tuning on robot demo data can impart a wide range of language-conditioned behaviors [1–6]. However, deploying these VLA models on a real physical system runs head-long into resource constraints. A transformer backbone with millions or billions of params demands a fast GPU to run at even a few frames per second. In many robot settings (mobile bases, consumer GPUs, even CPU-only microcontrollers), that’s impractical. As the AnoleVLA paper notes, memory and compute at inference time become the limiting factors when you try to get a VLA working on a robot. In one example, the authors point out that typical VLAs unify vision, language, and state into one long sequence of tokens, and then run it through layers of self-attention. Since attention cost goes like L², even moderately large images or extended language instructions can explode compute needs. Let’s compare some real numbers. The paper mentions a few existing approaches:. OpenVLA is an example VLA built on Meta’s Llama-2 (7B parameters). It achieves solid manipulation success rates in lab settings, but it’s heavy. π₀.₅ is a recent model from the Physical Intelligence team (CVPR 2025 TLDR). It uses a 3B-parameter LLM plus a separate “action expert” component. It also represents actions as discrete tokens (FAST tokens) and uses flow matching to convert them into real velocities. The result is strong long-horizon performance, but again requires huge compute and a multi-stage pipeline. On the other end of the spectrum, several groups have built smaller VLAs to reduce the footprint. For instance, the VLA-Adapter work shows that with only a 0.5B-parameter vision-language backbone (plus a tiny policy head), you can still get state-of-the-art success rates on a standard benchmark, and train it in hours on a single consumer GPU. Similarly, TinyVLA packs just ~0.42B into its model; it uses diffusion policy decoding and “robust high-speed multimodal” encoders, and it matches or beats a big OpenVLA model in both performance and speed. And SmolVLA (2025) is another compact design – on the order of hundreds of millions of parameters – explicitly built for affordable robotics. SmolVLA claims it can be trained on one GPU and run on a laptop CPU, yet still performs comparably to VLAs 10× larger. These prior efforts teach us that heavy VLMs aren’t strictly necessary, but they still typically use transformer-style backbones or multi-stage pipelines. AnoleVLA takes a different path: it retains a multimodal, end-to-end approach but swaps out the transformer core altogether for a lightweight recurrent model. The result, as we’ll see, is an incredibly efficient policy that nonetheless understands vision and language well. Deep State Space Models: The Linear-Scaling Alternative. To understand AnoleVLA, we need to briefly cover the idea of deep state-space models (SSMs). In contrast to transformers, SSMs process sequences by evolving a hidden state over time. At each new token, the model updates a fixed-size state vector using linear operations (like filtering with matrices). This implies a forward pass that costs only O(L) for a sequence of length L, rather than O(L²) as in self-attention. Classical state-space representations (think Kalman filters, or control-theory state equations) have been extended to deep learning. Recently, papers like S4 and Mamba have shown that carefully parameterized state-space layers can match transformers on long-context tasks. For example, Mamba (Meta AI, Dec 2023) introduced a selective SSM: it adds gates that let the model decide how much new input to integrate versus how much past state to carry forward. In concrete terms, a Mamba layer keeps matrices (A_n), (B_n), (C_n) and a selection gate at each block. As each token (h_{n-1,i}) arrives, Mamba updates its state (z_{n,i} = A_{n,i} z_{n,i-1} + B_{n,i} h_{n-1,i}) and outputs (\tilde h_{n,i} = C_{n,i} z_{n,i}) plus some gating and feed-forward. The net effect is that new tokens “filter” into the hidden state, which carries information from all prior steps. Crucially, this is a recurrent process, so each update is constant-time per token. In the words of the paper, “Mamba is a selective deep SSM… which allows the model to dynamically propagate or forget information along the token sequence. We adopt Mamba as the backbone… while keeping the same multimodal tokenization”. In robotics terms, this means the model can keep a running summary of the scene and instruction that gets refined each timestep, without paying the cost of recomputing attention over all tokens. Deep SSMs are already showing up in robotics literature. For example, “RoboMamba” (2024) used an S4 backbone to transform camera and proprioceptive inputs into predicted contact points and end-effector goals. FlowRAM (2024) embedded Mamba blocks within a diffusion-policy scheme, blending images and state before generating actions. There are also works on SSM-based diffusion policies and proprioceptive control. These suggest that an SSM can serve as a fast policy core if designed right. AnoleVLA leverages this by using a Mamba SSM stack in place of the usual transformer. The sequence it processes is still vision tokens + language tokens + robot state tokens, but they flow through the Mamba layers linearly. Each SSM block applies its learned filter (matrices (A, B, C)) and a small feed-forward. Because the hidden state is updated token by token, the model naturally aligns with a causal, incremental processing of the scene: early tokens (like proprioceptive state) influence the state first, then image features, then language. The authors even note that they place the robot state tokens at the beginning of the sequence so that “the hidden state is first conditioned on the agent’s state before integration of visual and linguistic information”. This ordering leverages the Mamba recurrence: it’s like feeding the robot’s egocentric state in, then letting the images and text modify that state as they arrive. By switching to Mamba, AnoleVLA’s sequence processing has linear (O(L)) cost. In practice this yields much lower memory usage and faster throughput. The authors explicitly cite that “Transformer self-attention scales quadratically … [and] as the number of vision tokens or temporal context grows, this quadratic cost can be a critical bottleneck… To overcome this, we propose AnoleVLA … that leverages a deep state space model (deep SSM) for efficient multimodal sequence modeling.”. And indeed, in experiments, this choice translates into about 3× faster inference on GPU and much lower subject latency than a comparable transformer VLA. The AnoleVLA Architecture. Putting the pieces together, AnoleVLA’s network has three main components: (1) a multi-stream embedding layer, (2) a stack of Mamba deep-SSM blocks (the backbone), and (3) a simple action decoding head. Inputs: At each decision timestep (t), the robot’s input (x^{(t)}) consists of four parts: the current state (s^{(t)}) (e.g. joint positions or gripper pose), the change in state (\Delta s^{(t)} = s^{(t)} - s^{(t-1)}), the camera image (x_v^{(t)}), and the language instruction (x_\ell). The state and delta-state get embedded to two single-token vectors via linear layers. The image is first encoded by a pre-trained SigLIP-2 vision encoder (a CLIP-like model for images) and then linearly projected into a sequence of visual tokens ({h^v_{t,1},\dots,h^v_{t,N_v}}). The language is tokenized and also projected into a sequence ({h^\ell_{1},\dots,h^\ell_{N_\ell}}). These four streams of embeddings are concatenated into one long input sequence: [ H^{(0)} = [,h^s,; h^{\Delta s},; h^v_{1:N_v},; h^\ell_{1:N_\ell},], ] with a total length (L = 2 + N_v + N_\ell). (Here (h^s) is “state”, (h^{\Delta s}) is the delta, the (h^v_i) are vision tokens, and (h^\ell_i) are language tokens.) The authors deliberately put the state tokens first so that the recurrent Mamba layers immediately see the robot’s own state, then gradually incorporate vision and language. SSM Backbone (Mamba blocks): The concatenated token sequence is fed into a stack of K Mamba blocks. Each block is a selective SSM layer followed by a small feed-forward (a SiLU-gated linear unit) and normalization. In each block, tokens are processed one by one: the hidden SSM state (z_{n,i}) at token position (i) is updated from the previous state (z_{n,i-1}) and the token input (h_{n-1,i}) as [ z_{n,i} = A_{n,i},z_{n,i-1} + B_{n,i},h_{n-1,i}, \quad \tilde h_{n,i} = C_{n,i},z_{n,i}, ] where (A,B,C) are learned matrices (potentially gated or adaptive by token). The output (\tilde h_{n,i}) goes through the rest of the block (nonlinearity, residual, etc.) to produce the block output (h_{n,i}). Because each block processes tokens in order (causally), the final hidden state of token (i) has integrated all earlier tokens in the sequence. Over K layers, the model progressively refines a running “summary” of the scene and instruction in its final token’s state. As the authors put it, the SSM update “continuously refines a running estimate of task-relevant information as new tokens arrive, aligning naturally with robotic manipulation where the model incrementally integrates visual observations to determine actions”. Action Head: After the Mamba stack, we take the last token’s embedding vector (i.e. the output of the final layer for (i=L)) and feed it through a simple linear projection to produce the action. In AnoleVLA, actions are represented as a fixed-length chunk of future velocity commands (e.g. 50 time steps), (\hat y_{t+1:t+H}). Concretely, they learn a linear map (f_{\text{head}}) such that [ \hat y_{t+1:t+H} = f_{\text{head}}(h_{K,L}), ] where (h_{K,L}) is the last output token of layer K. Because the Mamba hidden state has already aggregated all multimodal info up to (t), a linear head suffices to decode the needed velocity controls. This head outputs continuous 6-DoF manipulator velocities (or end-effector pose deltas) over the next window. Importantly, this means AnoleVLA never discretizes actions into language tokens – it directly outputs real-valued motor commands. By contrast, many earlier VLAs forced a discretization (tokenization) of continuous control and then used something like flow-matching or diffusion to recover smooth actions. AnoleVLA’s direct-output design is simpler and avoids quantization artifacts. Putting it all together, AnoleVLA is a single neural network that consumes (state, delta, image, text) and produces (velocity trajectory). Its core distinction is that the “backbone” is an SSM (Mamba) rather than a transformer. The model size ends up around 0.47 billion parameters, roughly comparable to the other compact VLAs. But thanks to linearized attention and a streamlined output, it achieves extremely high inference speed and low memory use. The authors summarize their novelties as: a lightweight, efficient VLA for resource-constrained use; the Mamba-based backbone for efficient sequence modeling; and a two-stage training strategy with an acceleration loss. Two-Stage Training: Velocity and Acceleration Losses. Training the policy follows self-supervised imitation learning on demonstration data. During the first phase of training, AnoleVLA is taught to predict the ground-truth end-effector velocities (or delta poses) at each future timestep. In other words, the linear output (\hat y(\tau)) (for (\tau=1\ldots H)) is matched against the demonstrated velocity (y(\tau)). This stage uses a standard regression loss on velocities. After this initial training, the network can produce reasonable motions, but they might be jerky or non-smooth. To refine the quality of actual execution, the authors introduce a second training phase with an acceleration or smoothness loss. Specifically, they encourage the difference between successive velocity predictions to match the true differences in the expert trajectories: (\Delta \hat y(\tau) = \hat y(\tau) - \hat y(\tau-1)) should align with (\Delta y(\tau) = y(\tau) - y(\tau-1)). Intuitively, penalizing the second derivative of position (the acceleration) leads the network to favor smoother control signals. In practice, this two-stage scheme has a noticeable effect. An ablation in the paper compares a model trained without the acceleration loss (just velocity loss) to one with the two-phase training. The “with acceleration” model saw its average success rate rise from about 63.1% to 67.85%. The authors remark that adding the acceleration objective “enabled the generation of smoother and more stable trajectories, which effectively improved overall performance”. This makes sense: in a control context, a slightly phasing trajectory that is stable often has a better chance of actually catching and moving an object than one that jerks around. We emphasize that this training trick is fairly lightweight – it just means adding a secondary loss term – but it pays off in practice. Empirical Results. AnoleVLA was tested on both simulation benchmarks and real robot experiments, using a variety of manipulation tasks. Quantitatively, it consistently outperformed all baselines, including some much larger models, while running faster. Figure 1 of the paper (reproduced conceptually below) shows the tradeoff:. On a 2D scatter plot of success rate versus inference speed (Hz), AnoleVLA sits at the top-right corner. Its success rate averages over tasks is the highest among models, and its inference speed is roughly 3× that of a representative large VLA (the π₀.₅ baseline). Numeric results: on the test suite considered, AnoleVLA achieved about 67.9% average task success, compared to 57.3% for SmolVLA (the best-speed competitor) and only 43.8% for π₀.₅. That means AnoleVLA outperformed SmolVLA by +10.5 percentage points, and beat π₀.₅ by +24.0 points. All that with a model size (~0.47B) far smaller than the 3B of π₀.₅. (For reference, VLA-Adapter and TinyVLA – other compact models – scored much lower on these tasks: e.g. TinyVLA ~31.6% and VLA-Adapter only ~0.9% in this evaluation.). Real-world deployment: Perhaps more impressively, on a physical robot the trends held. The authors report that in real experiments (across the suite of tasks), AnoleVLA’s success rate was 21 points higher than the large baseline, while running nearly three times faster. In real time, this means a dataset where SmolVLA or π₀.₅ might run at 5–10 Hz, AnoleVLA runs at 15–30 Hz and completes tasks more often. This reduction in latency is critical if you want smooth control and responsiveness on an actual robot platform. Qualitative examples underscore these numbers. The paper describes two scenarios (see Fig. 4):. Task: “Pick and place a puck onto a shelf.” At (t=0), the scene has a blue puck and a wooden shelf. AnoleVLA’s policy successfully steers the arm, grasps the puck, and places it on the shelf. By contrast, the SmolVLA baseline approaches the puck but fails to grasp it – instead it sort of drifts and moves toward the shelf, ultimately failing the task. This suggests AnoleVLA was better at aligning and grasping the object under the language goal. Task: “Hammer a screw on the wall.” The initial scene has a hammer and a stationary screw. AnoleVLA correctly grasped the hammer and swung to strike the screw exactly. The baseline, however, did grasp the hammer but then swung it poorly and missed the screw. In other words, AnoleVLA generated a more precise tool-use trajectory. One failure case is noted: in a “push the puck to a goal” task, AnoleVLA sometimes fails to make proper contact with the puck. So it isn’t perfect – pushing can be tricky and small errors accumulate. But overall, the qualitative behavior shows that replacing the backbone with Mamba did not harm language understanding or motor precision; if anything, the policy’s trajectories became more reliable and fluent (especially after the acceleration loss finetuning). Discussion. Several key takeaways emerge from AnoleVLA:. Efficiency with Performance: By shifting to a state-space backbone, AnoleVLA achieves significant savings in compute and memory. The authors emphasize that the Mamba-based model has linear sequence complexity, making it far more amenable to long contexts in realtime. At the same time, the policy performance did not drop – it actually improved relative to heavier baselines on the tasks tested. This suggests that, at least for manipulation tasks, the bottleneck was not the expressivity of transformers but simply their overhead. AnoleVLA shows we can “have our cake and eat it too” by choosing a lighter backbone. Direct Continuous Actions: Most existing VLAs emit discrete tokens and then employ a further step to turn them into smooth controls. AnoleVLA breaks that mold by predicting raw velocities directly. This means no quantization error and a simpler training objective. It also lets the policy exploit the fact that actions in robotics are inherently continuous – the model doesn’t have to pretend they are words. The two-stage velocity+acceleration training further refines this, enforcing that the control signals vary smoothly in time. In effect, AnoleVLA serves directly as a low-level controller. Model Size vs. World Knowledge: One might worry that shrinking the backbone removes the large-scale pretrained knowledge from a big LLM. However, AnoleVLA still benefits from warm-starting parts of its encoders (e.g. SigLIP for vision). It then fine-tunes end-to-end on robotics data. The results imply that the model has enough capacity (and enough priors in the vision modules) to learn the needed multimodal grounding. In practice, a 0.47B Mamba can absorb plenty of task-relevant information, at least in the narrow domain of tabletop tasks. This aligns with the SmolVLA findings: a well-designed few-hundred-million-param model can “match VLAs 10× larger”. Recurrent State and Long Context: AnoleVLA’s hidden state carries information forward through time and space. This is beneficial in a dynamic environment or for longer-horizon tasks. Because the SSM is causal, it can naturally integrate vision frame by frame without having to re-attend to every past frame. For instance, if the robot has already seen the red bowl a few steps ago, the hidden state knows about it without reprocessing the entire image. This may allow AnoleVLA to effectively use a longer history of observations than a transformer could in realtime. Real-World Viability: Perhaps the most important practical point is that AnoleVLA seems ready for on-board robotic deployment. Its inference is fast enough to run at tens of Hertz on commodity GPUs (or even maybe embedded systems). Memory usage is low enough to fit on mid-range cards. This opens the door to using sophisticated VLA-style policies outside the lab. For example, a home or hospital robot equipped with this model could process vision-language commands on-device, without needing a server. In contrast, many transformer-based VLAs require cluster GPUs just to run. Looking ahead, this work suggests that state-space architectures deserve more attention in robotics. We see parallels with how classic control theory values internal state estimation – deep SSMs are effectively learning their own “internal dynamics” about the world. As multi-modal models grow, it will be essential to keep an eye on efficiency. AnoleVLA is one step in that direction, blending modern multimodal learning with resource-friendly sequence modeling. Conclusion. In summary, AnoleVLA introduces a compelling recipe for embodied AI on cheap hardware. By swapping out the quadratic-cost transformer for a linear-cost Mamba SSM, the model remains powerful enough to understand vision-and-language instructions for manipulation, yet trims inference latency and memory use drastically. It directly outputs smooth continuous controls, trained with a two-stage loss to refine motion. Empirically, this lightweight model outperforms or matches bulkier baselines on pick-and-place and tool-use tasks, while running roughly 3× faster. Put simply, AnoleVLA shows that high-performance VLAs need not be locked into massive transformers: lean, specialized architectures can take us a long way. This work will likely inspire further use of state-space methods in robotics. It also demonstrates that we can adapt ideas from natural language processing to robotic control – in this case, leveraging the latest sequence models (S4/Mamba) to unlock practical VLA deployment. For practitioners, the takeaway is clear: if you want your language-conditioned robot to run on a small machine, consider architectures like AnoleVLA. And for the research community, this report underlines the importance of algorithmic efficiency as we build increasingly capable robot “brains.”. References: The above discussion is based on the AnoleVLA paper by Takagi et al. (ArXiv 2026) and related works, including π₀.₅, VLA-Adapter, TinyVLA, and SmolVLA. Key technical points about deep state-space models come from the description of Mamba in the AnoleVLA architecture and the ablation results. All quoted performance numbers are from these sources.