Latest / Embodied AI 101 / DoRA: Weight-Decomposed Low-Rank Adaptation

Transcript

DoRA: Weight-Decomposed Low-Rank Adaptation · Embodied AI 101

• DoRA: Weight-Decomposed Low-Rank Adaptation. Parameter-efficient fine-tuning (PEFT) of large pre-trained models – especially large language and vision-language models – has become a standard tool for practitioners. Methods like LoRA (Low-Rank Adaptation) freeze the vast majority of a model’s weights and only adjust a small low-rank modification, dramatically reducing memory and compute needs during training. This keeps inference exactly as before (no extra runtime cost) while still letting the model adapt to new data. LoRA saw the light of day in the CLIP and LLM world and quickly became a default choice for fine-tuning ever-larger models on a budget. Despite its practical success, users often observe that LoRA-style tuning can fall a bit short of full fine-tuning performance. In other words, LoRA-trained models sometimes lag behind a fully fine-tuned model on the final metrics, especially on harder tasks. The new DoRA method – short for Weight-Decomposed LoRA Adaptation – was proposed to close exactly this gap. In an ICML 2024 oral paper, Liu et al. (NVIDIA Research and HKUST) carefully analyze why LoRA tends to underperform full fine-tuning, and design a tweak that gives LoRA nearly the same flexibility as full tuning while keeping inference free of overhead. The key insight is to break each weight update into two orthogonal parts: the direction of the weight vector and its magnitude (length). DoRA explicitly learns both, using a LoRA-style low-rank update only for the directional component and a simple scale factor for the magnitude. This double tracking of weights (“direction vs magnitude”) dramatically increases LoRA’s learning capacity and stability. In practice DoRA comes as a drop-in upgrade: after training one can merge the learned components back into the weight with no extra cost at inference. DoRA consistently outperforms LoRA on many benchmarks, from language understanding to visual instruction tasks, often by several percentage points. The improvement is especially striking when quantization is used: a quantized DoRA (“QDoRA”) can even beat full fine-tuning on some benchmarks while using a fraction of the memory. With easy support in popular libraries (Hugging Face’s PEFT, NVIDIA’s code, etc.), DoRA is poised to become a new default for fine-tuning LLMs and VLMs. Below we unpack how DoRA works, why it helps, and what gains it brings. We assume familiarity with LoRA and PEFT methods, but will explain the new concepts of weight direction and magnitude. We won’t dive into heavy math – instead we describe the ideas in plain terms – although this work does connect to classic weight normalization techniques in deep learning. By the end you should understand why DoRA improves upon LoRA, how it is implemented, and how to use it in your own models. Background: Parameter-Efficient Fine-Tuning and LoRA. Large pre-trained models (LLMs, vision-language backbones, etc.) are often fine-tuned on downstream tasks. Full fine-tuning – allowing every weight to update – yields the best accuracy, but costs huge memory and compute, especially for today’s 10B+ parameter models. For example, even a moderately large Llama model can require hundreds of gigabytes to store gradients and optimizer state. Parameter-efficient fine-tuning (PEFT) methods were introduced to reduce this cost, by freezing most of the model and only training a small number of additional parameters. These methods include prompt/prefix tuning, adapters, and low-rank adaptation (LoRA), among others. LoRA in particular has become very popular for Transformer-based models. The core idea of LoRA is simple: fix the original weight matrix $W$ and add a trainable low-rank update. Formally, one replaces a weight matrix $W_0$ with$$ W = W_0 + \Delta W,\quad \Delta W = B A, $$where $A$ and $B$ are small matrices of dimensions ($r \times d_{\text{in}}$) and ($d_{\text{out}} \times r$) respectively. Here $r$ (rank) is tiny (often 4–32) compared to the dimensions of $W_0$, so $B$ and $A$ only add a small fraction of extra parameters. In effect, each forward pass computes $W_0 x + (B (A x))$ in place of the usual $W_0 x$. Because $W_0$ is frozen, the extra term can be merged at inference (so inference uses a single effective weight matrix). The training is done via backpropagation updating just $A$ and $B$ (and possibly a scaling factor). LoRA can drastically reduce the memory for gradient updates, lowering it by a factor of dozens or hundreds, while still capturing much of the adaptation needed for many tasks. It was originally proposed for NLP models (“$Q$, $K$, $V$ weight tuning”) and has since been extended widely. The Hugging Face PEFT library, diffusers, and other toolkits now have built-in LoRA support, reflecting its utility. With all these advantages, you might wonder: why bother with anything beyond LoRA? The answer is that empirically, LoRA sometimes underperforms full fine-tuning. In practice, for many tasks LoRA matches or nearly matches the fully fine-tuned model, but on some harder benchmarks it can fall behind. For instance, AnyScale’s in-depth analysis of Llama-2 fine-tuning found exactly that: LoRA often comes close to full fine-tuning, but there are noticeable gaps on several tasks. The NVIDIA team (Liu et al.) also observed this pattern. This performance gap can be crucial in high-stakes settings (e.g. safe reinforcement learning, difficult vision-language reasoning) or simply in squeezing out top accuracy. Moreover, if one wants to train very large models on limited hardware (e.g. consumer GPUs), one typically uses quantized LoRA approaches (QLoRA) – yet the LoRA limitations carry over there too. In short, LoRA is a great step toward memory efficiency, but it has a limited capacity compared to full fine-tuning. The question DoRA asks is: what exactly is LoRA not capturing that full tuning does? By analyzing this, they design a modified adaptation that covers the missing capacity. Weight Decomposition: Magnitude vs Direction. The key technical insight of DoRA is that any weight vector (or matrix) in a neural net can be thought of in polar terms: it has a direction (a unit vector) and a magnitude (a scalar length). A simple analogy is helpful: imagine each row of a weight matrix $W_0$ as a point in high-dimensional space. Full fine-tuning can move that point anywhere: it can rotate it in a new direction and change its distance from the origin. In contrast, LoRA’s update can also rotate the weight vector, but it implicitly entangles magnitude and direction changes together in a fixed pattern determined by $B A$. More concretely, Liu et al. performed a weight-decomposition analysis. They examined how weight updates in full fine-tuning (FT) compared to updates in LoRA. For each weight vector, they split the update $\Delta W$ into two parts: $\Delta W_{\text{dir}}$ which is parallel to the original weight (magnitude change), and the component orthogonal to it (direction change). Plotting these two components against each other for FT vs. LoRA revealed a striking pattern. In full fine-tuning, changes in direction and magnitude tend to be decoupled: if the direction update is large, the magnitude change can be small, and vice versa (they saw a roughly negative correlation). In LoRA’s case, however, direction and magnitude often rise and fall together (a positive correlation). Intuitively, LoRA’s low-rank addition tends to move weight vectors in a correlated way along both axes rather than independently. This means LoRA lacks the freedom to, say, rotate a weight vector without also proportionally stretching it, or to scale it up without a large directional shift. In contrast, full tuning can do each separately. DoRA’s goal is to mimic full fine-tuning’s flexibility: it must allow a weight to change direction without being forced into a particular magnitude change. In the words of the Answer.AI team (which built on DoRA for quantized training): “Full fine-tuning independently optimizes the direction and magnitude of weight updates, whereas LoRA only updates them in tandem. DoRA is the best of both worlds: it updates them independently, like normal FT, but with the parameter efficiency of LoRA”. In short, DoRA explicitly gives LoRA an independent knob for magnitude. This insight – splitting weight updates into magnitude and direction – is reminiscent of the classic deep learning technique weight normalization. Weight normalization reparameterizes a weight as a scalar g (magnitude) times a unit vector v (direction). DoRA applies a similar idea to LoRA: it keeps LoRA’s low-rank structure for proposing a new direction, and introduces a learnable scale factor to fix the magnitude. The DoRA Method: How It Works. Concretely, DoRA works like this. Consider a linear or attention weight matrix $W_0 \in \mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ that is frozen (as in LoRA). We want to finely adjust this weight. We introduce two trainable components:. A magnitude vector $m \in \mathbb{R}^{d_{\text{out}}}$ (one scalar per output dimension). Initially, $m_i$ is set equal to the original length $|W_{0,i}|$ of row $i$ of $W_0$ (so the effective weight starts the same). During training, each $m_i$ will be adjusted by gradient descent (it’s a normal parameter). A LoRA-style low-rank update $(A,B)$ of rank $r$, exactly as in standard LoRA. That is, $A \in \mathbb{R}^{r\times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}}\times r}$ are trainable (usually $r$ is small like 8 or 16). As usual, the LoRA output is $B (A x)$ for an input $x$. Now, when we do a forward pass on input $x$, DoRA proceeds in two stages. First it computes the usual LoRA-suggested new weight vector: $$ \widetilde{W} = W_0 + \Delta W,\quad \Delta W = B A \quad(\text{i.e. } \widetilde{W}i = W{0,i} + (B A)_i \text{ for each output row }i). $$ Here $\widetilde{W}$ is what you would normally get from LoRA alone. Next, instead of using $\widetilde{W}$ directly, DoRA rescales it to impose the learned magnitude $m$. For each output row $i$, compute the current norm of $\widetilde{W}i$ (treating it as a vector in $\mathbb{R}^{d{\text{in}}}$) and compute a scale factor: $$ \text{norm_scale}_i = \frac{m_i}{|\widetilde{W}_i|}. $$ Then the effective weight used is $m_i \cdot \frac{\widetilde{W}_i}{|\widetilde{W}_i|}$, which indeed has length $m_i$ and points in the direction of $\widetilde{W}_i$. In code this is done by scaling the output of the linear layers: one can fold the scaling into the base and LoRA parts. The result is that the final output of the layer is$$ m_i\frac{\widetilde{W}_i}{|\widetilde{W}_i|} x ,, $$ for each output row $i$. Equivalently, one first computes $W_0 x$ and $BAx$ as usual, then multiplies their sum by the scalar $\text{norm_scale}_i$. In practice, one implements this by computing $\widetilde{W}$, then applying a per-row scaling. The authors’ PyTorch-like pseudocode shows exactly this: after computing , they compute and then apply to the base output and to the LoRA output, effectively scaling both appropriately. Because $\text{norm_scale}$ can be precomputed per row, this adds effectively one extra multiply per output in the forward pass, which is negligible. Importantly, neither $W_0$ nor $BA$ is kept trainable. We still freeze the original large $W_0$ (as in LoRA), update only the low-rank parameters $A,B$, and update the new magnitude vector $m$. The total number of trainable parameters is thus $(d_{\text{out}} \times r + r \times d_{\text{in}}) + d_{\text{out}}$, i.e. essentially LoRA’s cost plus $d_{\text{out}}$ more. For large matrices this extra $d_{\text{out}}$ is small compared to the usual LoRA terms (often LoRA parameters are in the hundreds of thousands or millions, whereas $d_{\text{out}}$ might be only a few thousand). In practical terms, you pay only a few percent more parameters than LoRA, but gain much more flexibility. Once training is finished, we can merge everything back into a single weight matrix for inference. Concretely, the effective final weight is $W_{\text{final}} = \operatorname{diag}(m)\cdot\frac{(W_0 + BA)}{|W_0 + BA|}$ (row‐by‐row norm). Compute and store this $W_{\text{final}}$ (and drop the LoRA and $m$ variables), and inference proceeds with a normal linear or attention layer. This means DoRA does not slow down inference at all. You get a single adjusted weight matrix of the same size as before, so run-time latency is identical to the base model. In NVIDIA’s words: “DoRA … avoids any additional inference overhead”. In summary, DoRA is a two-part modification: it uses a LoRA update for direction plus a learned scale for magnitude. By construction it has roughly the same memory profile as LoRA training (with a small extra vector) and zero extra cost at inference. This is why one can literally “flip a switch” from LoRA to DoRA in libraries like Hugging Face’s PEFT package: just set in a LoRA config, and it will automatically add the extra scale parameters and norm-scaling logic. Capacity Gap: Why Decoupling Helps. Why does this decomposition help so much in practice? The high-level reason is that it gives the optimizer a new degree of freedom that was missing in plain LoRA: namely, it can tune how large a weight vector is, separately from which direction it points. We’ve mentioned the visualization of weight updates, but let’s flesh out what that means algorithmically. When fine-tuning normally (all weights trainable), each weight vector $W_{0,i}$ can be nudged in any direction and scaled up or down freely. If a particular task demands strengthening certain features, the optimizer might decide to boost the magnitude of some weight, or to partially “flip” it, etc. LoRA’s constraints can hinder that. Recall, LoRA’s lowrank update $BA$ is basically chosen to approximate the true gradient update under rank-$r$ restriction. But that rank-$r$ structure tends not to perfectly align with how one might want to scale $W_{0,i}$. In effect, LoRA often ends up increasing the magnitude of $W_{0,i}$ automatically when its direction is being adjusted. That coupling can make training less stable or less precise. DoRA explicitly addresses this. For example, if full fine-tuning wanted to keep the current weight direction but just amplify it, DoRA can do that by increasing $m_i$ and keeping $BA$ small (or aligned with $W_{0,i}$). Conversely, if the task requires flipping the pattern of connections (direction) without changing norms, DoRA can update $BA$ to tilt $W_{0,i}$ and then set $\text{norm_scale}$ so that the length stays near $m_i$ (rather than ballooning). These moves are much harder in plain LoRA. The answer.ai blog illustrated this elegantly at a high level. They note that DoRA is essentially “inspired by Weight Normalization” and that it “splits LoRA layers into two components, one for magnitude and one for direction, and finetunes both”. Importantly, this makes DoRA better match the “capacities of full fine-tuning” without blowing up parameter count. In practice, during training we often see that DoRA converges a bit slower than LoRA (since it’s wandering a larger space), but the eventual quality is significantly higher, especially at modest ranks. In terms of optimization, you can think of DoRA as adding a normalization step after each LoRA update. LoRA proposes a new weight $\widetilde{W}$, then DoRA projects that back onto a sphere of radius $m$. The radius itself is learned. Conceptually, this is very similar to “weight normalization” in SGD optimizers, which often accelerates and stabilizes training by explicitly controlling norms. Here it has the side-effect of increasing expressive power of the low-rank update. Liu et al. note that DoRA not only boosts capacity but also “enhances … training stability of LoRA” – presumably because decoupling reduces pathological gradient conflicts. Empirical Results: Better Fine-Tuning Performance. The DoRA paper (and accompanying code) demonstrates consistent gains across a wide range of tasks and models, compared to the corresponding LoRA baseline. We summarize some highlights here. Language Comprehension and Commonsense Reasoning. A centerpiece of the evaluation is fine-tuning LLaMA-family models on commonsense reasoning benchmarks (BoolQ, PIQA, SIQA, HellaSwag, Winograd, ARC-easy, ARC-challenge, OBQA). In every case, DoRA gave a higher average score than LoRA. For instance, with a 32-rank LoRA configuration on LLaMA3-8B, the LoRA-tuned model averaged about 80.8% on these tasks, whereas DoRA (also rank 32) averaged 85.2% – a large +4.4 percentage point jump. Even at half the rank (r=16), DoRA already surpassed LoRA’s full-rank result: LLaMA3-8B with LoRA (r=32) got 80.8, but DoRA (r=16) got 85.0. Similar trends held for LLaMA-7B and LLaMA2-7B: in each case DoRA’s average was noticeably higher. For example, LLaMA-7B’s average went from 76.3% (LoRA, r=32) up to 78.4% or 77.5% with DoRA (r=32 or r=16), and LLaMA2-7B’s from 77.6% to around 80.5%. These are not isolated improvements: every individual task saw DoRA match or beat LoRA, so the gains aren’t due to a lucky artifact of one setup. Another metric was multi-turn conversational instruction following (MT-Bench scored by GPT-4). Here too, DoRA edged out LoRA on both LLaMA and LLaMA2 backbones (e.g. ~5.5 vs 5.1 for Llama, ~6.0 vs 5.7 for Llama2 on a 10-point scale). In summary, across standard NLP tasks, DoRA’s improvements ranged from a few tenths of a percent to several percent, consistently favoring DoRA. Vision-Language and Instruction Tasks. DoRA’s benefits extend beyond pure text. The authors also fine-tuned vision-language models on standard benchmarks. For example, on VL-BART (an image/video-question answering model), DoRA improved average scores modestly (from 76.5 to 77.4 on image-text tasks, and from 83.5 to 85.4 on video-text tasks). In the visual instruction tuning of LLaVA-1.5 (a 7B visual chatbot), DoRA gave a slight bump in average score (66.9 → 67.6). These gains are smaller than the language-only ones, but still consistent. They indicate that the extra flexibility in weight updates helps in multi-modal contexts too – perhaps by allowing the vision-language model to learn more nuanced correlations or to better calibrate attention. Using Quantization (QDoRA). One of the most impressive results is in the quantized setting. QLoRA showed that one can quantize a model (to 4-bit, say) and still train LoRA updates, saving huge memory. The DoRA authors explored “QDoRA”: essentially the same approach but applied after quantization of the backbone. According to NVIDIA/Answer.AI experiments, QDoRA not only surpasses quantized LoRA (QLoRA) but can sometimes even exceed the accuracy of full-precision fine-tuning. For example, in training on the Orca-Math dataset, a 4-bit QDoRA Llama3-8B greatly outperformed a 4-bit QLoRA and even matched or beat full-finetuned Llama3 (which of course used far more memory). In fact, Answer.AI reports: “Llama3 with QDoRA (or with full finetuning) greatly outperform [sic] QLoRA [on Llama2] when training on mathematical data”. They also note that QDoRA is “a scalable and memory-efficient method to close the gap between efficient finetuning and full finetuning”. In simple terms, because DoRA narrows the gap between LoRA and full FT, even the quantized version preserves most of that advantage. Their graphs of loss curves show a sharper descent for QDoRA vs QLoRA, and their benchmarks on zero/few-shot accuracy demonstrate clear wins (see [49] and the QDoRA blog for details). Practically speaking, it means you can fine-tune a massive model on a couple of small GPUs via QDoRA and get results as good as (or better than) you would get by full fine-tuning on a TPU pod – all thanks to the extra expressivity DoRA provides. Quantitative Examples. To make the above more concrete, consider Table 1 from the NVIDIA blog (recreated in the code repo). For LLaMA-3 8B on commonsense tasks, the LoRA-tuned model averages 80.8% accuracy, whereas DoRA (r=32) reaches 85.2%. That +4.4 boost is huge by modern standards, comparable to what one might gain by jumping to a substantially larger model. On LLaMA-2 7B, LoRA scores 77.6% while DoRA hits 80.5%. And on Llama-7B, 76.3% → 78.4%. Importantly, these are all with the same number of trainable parameters (aside from a ~1% extra), so they are apples-to-apples. In Table 3 (image-text tasks) and Table 4 (video-text tasks) from the paper/blog, DoRA leads by roughly 0.5–1.5 points on average over LoRA. On visual instruction tuning (Table 5), DoRA edges out LoRA by about 0.7 points on average. These vision-related gains are smaller, but they show DoRA is capturing something LoRA missed even in multi-modal layers. Training Behavior and Stability. Beyond final accuracy, the authors also noted that DoRA tends to improve training stability. Because magnitude can now be adjusted, sometimes the model doesn’t blow up or collapse in the same way. Anecdotally, they suggest starting with a slightly lower learning rate and maybe lower LoRA dropout when using DoRA, because DoRA often converges a bit slower than LoRA (it has more freedom). But the end result is usually better. The code repository suggests even halving the rank compared to LoRA: one often gets better performance with DoRA(r=16) than LoRA(r=32), implying DoRA’s updates are more efficient in parameter usage. Using DoRA in Practice. From a user’s standpoint, DoRA can be as effortless as LoRA thanks to library support. NVIDIA and Hugging Face have already integrated DoRA into their tools. In the NVlabs GitHub repo, the developers advise that after installing the latest (Hugging Face PEFT) release, one can simply set the flag when creating a. This automatically creates the extra magnitude parameters and handles the norm-scaling under the hood. The rest of the fine-tuning code (optimizers, training loop) remains unchanged from a LoRA setup. When calling an LLM with DoRA weights, at inference time everything merges into a normal linear layer. DoRA’s code ensures that before passing the final weights downstream, the scaling is applied so that the frozen weight and low-rank update become one fused matrix. Therefore there is no special inference code needed – you just run the model like usual and it uses the updated weights. This also means that any framework (e.g. Triton optimizers, ONNX export) that worked for the base model will continue to work seamlessly. For multimodal tasks (vision-language), the implementation is similar. The DoRA GitHub and NV blog note that DoRA supports Linear, Conv1d, Conv2d layers, and even linear layers in quantized form with bitsandbytes. So you can finetune VL adapters or diffusion UNets with DoRA as easily as LoRA. On the hardware side, because DoRA adds only a tiny number of parameters, training memory is virtually the same as LoRA. Most of the memory is still in storing activations for $W_0$ which is fixed, so it doesn’t change. CPU/GPU memory for Gradients only needs to cover $A,B,m$, which is just slightly bigger than LoRA’s. This makes DoRA well-suited to the same “consumer GPU” or “cloud A100” scenarios where LoRA/QLoRA are used. In fact, Answer.AI demonstrated fine-tuning Llama 3 models on a couple of 24GB GPUs using FSDP+QDoRA – something that would be infeasible with full training. NVIDIA has also released pretrained “DoRA weights” for some of their experiments. For example, on LLaMA2-7B we have official LoRA and DoRA adapters for the commonsense tasks, so one can download a LoRA weight or its DoRA counterpart and compare. (The repo shows better metrics for DoRA versions.) This is handy for benchmarking: you could try a LoRA and DoRA weight on your validation set and see how they differ in behavior or outputs. Finally, DoRA requires very minor hyperparameter tuning. The authors suggest to usually start from the LoRA context: use the same rank $r$ and initial learning rate as you would with LoRA. However, they note that DoRA tends to converge a bit slower, so one might consider lowering the learning rate slightly or increasing training steps. Also, because DoRA can make larger final weights, dropout can sometimes be reduced. In practice, however, many users report that simply swapping with their existing LoRA setup already yields an accuracy boost, and then a few tuning runs can optimize from there. Related Work. DoRA sits in the landscape of PEFT methods that seek to maximize accuracy with minimal extra parameters. It is closest to LoRA, but there have been other variations and enhancements:. AdaLoRA dynamically allocates the rank of the LoRA update by pruning singular values, so that the rank can grow only where needed. It still uses the same additive structure (no separate magnitude). In contrast, DoRA still uses a fixed rank but adds magnitude learning. In principle one could combine them (Adaptive DoRA) by pruning within the direction component, but that hasn’t been explored here. Weight Normalization (Salimans & Kingma) and Layer Normalization are older techniques that explicitly separate weight norms from unit vectors. DoRA’s idea is akin to applying weight normalization to LoRA adaptation layers: the magnitude vector is like the g-parameter in weight-norm, and the direction is adjusted by LoRA. This connection was noted by the authors. VeRA (Vector-based Random Matrix Adaptation) is a contemporaneous ICLR’24 paper that also talks about vectorizing low-rank updates, though in a different sense. VeRA learns random base matrices that are scaled by vectors. DoRA’s approach of having a scaling vector $m$ is reminiscent in spirit, but DoRA’s scale is learned directly from data (with gradients) rather than fixed. In fact, the DoRA authors integrate with VeRA in their “DVoRA” variant for instruction tuning. Adapter Layers insert small bottleneck MLPs, but those usually increase inference cost. DoRA explicitly keeps inference cost down, so it’s an “addition-only” method like LoRA. Prefix Tuning / Prompt Tuning add new input tokens or continuous embeddings, which again is a very different mechanism. They do not alter the weights. DoRA’s category is the “additive weight update” family, so it complements prefix/prompt approaches. More recently (HydraLoRA, Progressive LLaMA (LLaMA-Pro) etc.) some methods alter the architecture or training schedule. DoRA’s contribution is much simpler and completely general-purpose: it can be applied to any linear layer in any Transformer. Some works like LoRA++ or BitFit tweak only biases or scales, but DoRA is distinct in that it still uses the LoRA $A, B$ structure plus a small extra vector. In summary, the unique thing DoRA brings is the explicit decomposition of weight updates into magnitude and direction, and training both. This pretty clearly draws on the intuition of weight-normalization, while staying very close to LoRA’s philosophy of minimal overhead. No other PEFT method to our knowledge does exactly this. It thus fills the conceptual gap identified by their weight analysis. When and Why DoRA Matters for Practitioners. For practitioners and researchers using large models – whether in robotics, embodied AI, or other domains – DoRA offers an easy win in fine-tuning. If you were planning to use LoRA to adapt a pre-trained model, consider switching to DoRA. The NVlabs/AnswerAI results suggest you’ll often get a healthier final model, for free. This is particularly relevant if your application is tricky or your dataset is limited. For example, if you are fine-tuning a multi-modal model (like LLaVA, BLIP, etc.) for robot vision-and-language understanding, the small gains from DoRA might translate to noticeably better real-world performance (understanding instructions, answering vision questions, etc.). Likewise, in LLM-based planning or reasoning tasks, the extra capacity of DoRA often shows up as better reasoning accuracy or more coherent outputs. In robotics, computational resources on-device are often at a premium. DoRA doesn’t cost extra at inference, so robots with widgets like video-capable LLMs can benefit without slower inference. Moreover, if you ever want to update a deployed model with new data, you can use DoRA offline to get those improvements with minimal server GPU costs. Even beyond robotics, think of DoRA as a superior “drop-in replacement” for LoRA whenever you would otherwise be fine-tuning a transformer. It’s already integrated in PyTorch/PEFT ecosystems, and studies show it can match full fine-tuning accuracy more closely. In practice, that means your model will adapt better to specialized tasks – the “cost” you pay in extra parameters is so small that it’s usually worth it. Conclusion. DoRA – Weight-Decomposed Low-Rank Adaptation – is a clever and practical refinement of the LoRA fine-tuning recipe. By explicitly modeling weights as length × direction and learning both, DoRA closes much of the gap between low-rank tuning and full model adaptation. It achieves better task accuracy and stability without breaking the “no extra inference cost” promise of PEFT. Empirically, it brings gains on language and vision tasks alike, and it plays well with quantization (QDoRA). Given its ease of integration and strong results, DoRA is likely to become a standard tool in the toolkit of anyone fine-tuning large models. If you’ve been using LoRA, it’s highly recommended to try DoRA with the same settings; you will almost certainly see an accuracy boost. The extra insight from DoRA is also valuable for research: it highlights how controlling weight norms can be key when updating frozen models, a concept that might inform future PEFT ideas. In summary, DoRA takes a mature idea (LoRA) and injects a simple yet powerful additional degree of freedom. The result is as immediate as a package install (PEFT flip of a switch) but as substantial as closing the performance gap to full fine-tuning. For roboticists and ML engineers working on embodiment tasks, this means tougher models can be fine-tuned more effectively even on limited hardware. The code and models are already out there – it’s a great time to make DoRA part of your fine-tuning workflow. References: The DoRA method and results are detailed in Liu et al., ICML 2024 (see also the authors’ and GitHub repo). For example, the authors write, “DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates… DoRA consistently outperforms LoRA on fine-tuning [various models]”. An analysis shows that “full fine-tuning independently optimizes the direction and magnitude of updates… LoRA only updates them in tandem. DoRA is the best of both worlds”. In experiments on LLaMA-family models, DoRA yields several-point improvements over LoRA. Quantized DoRA (QDoRA) further achieves even higher accuracy in low-bit settings. These sources and the DoRA code provide full details for implementation and results.