Latest / Embodied AI 101 / Can CriticGPT Replace Hand-Designed Rewards for Robot Manipulation?
Transcript
- Can CriticGPT Replace Hand-Designed Rewards for Robot Manipulation?. CriticGPT: A Multimodal Large Language Model as a Critic for Robot Manipulation • Yifu Yuan, Jinyi Liu, et al. • Tianjin University • Electronics • 2026. Can a multimodal language model watch robot videos and supply the reward signal you would otherwise engineer by hand?. CriticGPT makes that approach credible in a constrained simulation setting. It does not establish a general replacement for reward engineering. Its central idea is to turn video comparisons into preference labels, then train a separate reward model that can support reinforcement learning. Crucially, the training supervision originates in privileged simulator information—not in a model independently discovering what successful manipulation means. My take is that the architecture is more compelling than the strongest interpretation of its results. Separating an expensive video evaluator from a lightweight training-time reward function is sensible engineering. But three questions remain distinct: Can the evaluator rank recorded behavior? Can a smaller network preserve those judgments? And will optimizing that network produce the behavior we actually want?. The publication timeline also matters. The linked journal article appeared on October 8, 2026, while an earlier arXiv account dates to February 22, 2024. This is an early robot-feedback idea receiving a later journal treatment, not an entirely new October 2026 research direction. What the Paper Did. The training set contains 24,000 examples from sixteen Meta-World tasks; the test set contains 7,500 from five tasks. SAC policy checkpoints generate behavior of varying quality. Simulator success, rewards, and distances supply the ingredients for templated analyses and preferences. That construction is worth unpacking before discussing the language model. Imagine two attempts at pressing a button. One robot moves close to the button and stops. Another presses it briefly, then withdraws. A useful evaluator should prefer the second attempt, even if its final image shows the gripper farther from the target. Comparing complete attempts potentially provides a more meaningful learning target than comparing two terminal snapshots. But the quality of that learning target depends on the rule that generated it. Should successful completion always dominate speed? How should two unsuccessful attempts be ordered? Does moving closer deserve credit if the robot approaches from a direction that makes contact impossible? These are specification decisions, even when a script applies them automatically. The important distinction, therefore, is between removing privileged information from the evaluator’s inputs and removing privileged information from the supervision pipeline. The former is a useful capability. The latter would be a much stronger achievement. A reasonable interpretation is that this approach tries to make existing simulator knowledge portable: first express that knowledge through examples, then learn to recognize corresponding visual evidence. That could reduce the need to instrument every deployment environment. It does not make the original task specification disappear. Architecturally, CriticGPT adapts LLaVA-1.5: the vision encoder and language backbone remain frozen, while LoRA adapters and the visual projection are trained. The output includes an analysis and a preference. This is primarily an adaptation and systems-integration contribution. The useful question is not whether low-rank adapters constitute a new robotics architecture. It is whether the resulting visual-language interface produces supervision that is sufficiently transferable and sufficiently precise to improve control. There is also a distinction between generating an explanation and needing that explanation. A model might produce a convincing account of an attempt while its actual ranking relies on a simpler visual cue. Conversely, generating an analysis might help organize the prediction. To separate those possibilities, I would want a matched training comparison between preference-only outputs and analysis-plus-preference outputs—not just examples of fluent explanations. In the journal implementation, each clip contributes fifty frames. The downstream reward network is image-only, task-specific, and frozen during policy learning. The mathematical bridge is familiar from preference learning. PEBBLE, by Kimin Lee, Laura Smith, and Pieter Abbeel, learns rewards from comparisons between behavior segments. It sums predicted rewards within each segment, converts the difference between those totals into a preference probability, and trains that probability against the teacher’s choice. PEBBLE also couples preference learning to off-policy reinforcement learning and replay-buffer relabeling. For intuition, suppose the teacher prefers an attempt that opens a door over one that merely touches the handle. The reward learner is encouraged to give the successful segment a larger total. But that constraint does not uniquely determine which frame deserves credit. Many different frame-scoring functions could produce the same ordering. One function might correctly emphasize the door moving. Another might overvalue a particular arm configuration that happens to occur in successful recordings. Both can satisfy the training comparisons. That is why “dense reward” should not be confused with “dense ground-truth supervision.” The network produces a number at every observation, but its training target constrains the relative value of whole segments. The allocation of credit within those segments is learned indirectly. CriticGPT then uses the learned rewards with DrQ-v2 to train image-input policies. The language model is the preference teacher, not the robot’s action generator. As a design choice, that separation is attractive. A practitioner can potentially batch video comparisons offline and avoid autoregressive language generation inside the control loop. It also exposes useful interfaces for debugging: inspect the teacher’s preferences, inspect the distilled reward, and inspect the policy separately. The price is another opportunity for error. Even an excellent trajectory evaluator does not guarantee that its judgments can be represented faithfully by a smaller, single-observation reward function. Consider a hypothetical task where an object must be inserted and then removed. Two histories can end in nearly identical images but have different completion status. More fundamentally, summing independent frame scores cannot distinguish two sequences containing exactly the same frames in different orders. If order changes which attempt is preferable, an additive image-only student cannot represent that distinction perfectly. This is a structural issue, not an accusation that every Meta-World task requires memory. It tells us where this particular interface should be stress-tested before extending it to more complex manipulation. How Much Does the Video Critic Actually Understand?. The reported preference accuracy is 99.9% on the standard test set and 70.6% on Hard-Test; unadapted LLaVA-1.5 reaches 41.2% on the standard test. Tied reference comparisons are excluded. That is a substantial adaptation result. It also illustrates why the denominator matters more than the headline. A comparison benchmark can be easy when one attempt clearly succeeds and another fails far from the goal. Such pairs are useful for learning broad distinctions. They are less informative about whether an evaluator can resolve the small differences that matter once a policy has become competent. The harder set is therefore especially relevant to deployment. If the model is being used to improve an already reasonable controller, its queries may increasingly concern near-successes, partial contacts, recoveries, and subtle mistakes. Those are precisely the kinds of comparisons for which coarse visual recognition may stop being enough. I would not interpret the near-perfect standard score as evidence of equally reliable physical understanding. It is evidence of agreement with a particular automatically constructed reference on a particular comparison distribution. The contrast with unadapted LLaVA is useful, but it answers a limited question: does domain adaptation help this model handle this interface? It does not establish that a language model is necessary, or that this architecture is the best way to use the available supervision. A temporal encoder with a pairwise classification head would be an important control. So would a visual model trained directly on the same simulator-derived preferences. If either matched the ranking performance at substantially lower cost, the language-model component would need to justify itself through transfer, task conditioning, or other capabilities. The journal’s input ablation reports standard-test accuracy of 96% with ten frames, 60% with final frames only, and 51% without video. My reading is that visual coverage matters. I would be more cautious about treating this as proof of temporal reasoning. Removing frames changes several things simultaneously: the events visible to the evaluator, the opportunities to see an unoccluded object, and the input distribution relative to training. An evaluator that behaves largely like a bag-of-images classifier could also benefit from seeing more frames. The sharper experiment would preserve the images while changing their order. Shuffle them, reverse them, or construct pairs that achieve the same intermediate states but differ in whether progress is undone. Those interventions help distinguish “I saw evidence of contact somewhere” from “I understood the sequence of events.”. There is a practical efficiency question here too. Before adopting the full video input, I would compare uniform sampling with event-aware selection. If a short contact event determines success, increasing the frame count everywhere may be less effective than ensuring that the event is actually observed. That is a proposed extension, not a demonstrated property of this system. The human study uses three annotators and 300 pairs across three tasks. CriticGPT agrees with majority human judgments on 92%, versus 86% for the automatic template. This is a more meaningful check than template agreement alone because it introduces a different reference. Nevertheless, human agreement and simulator correctness should not be treated as interchangeable. Suppose the simulator records a tiny successful button displacement that is difficult to see. Humans might reasonably prefer a visually clearer but technically unsuccessful attempt. Alternatively, a simulator-derived ranking could overvalue distance reduction while people correctly recognize that another attempt makes more useful progress. Both are plausible disagreement mechanisms. The next useful artifact would be a disagreement audit: the videos, each annotator’s choice, the template’s choice, and the model’s choice. That would tell us whether the evaluator is correcting undesirable template behavior, adopting human perceptual limitations, or doing some mixture of both. The broader lesson is that an evaluator needs an explicit contract. Is it predicting benchmark success, visible progress, human preference, or expected usefulness for policy improvement? Those objectives overlap, but they are not identical. Does Better Ranking Produce Better Policies?. Policy evaluation uses sixteen tasks, five training seeds, 300,000 environment steps, and twenty evaluation episodes per seed. Final macro-average success is 61.6% ± 27.3 for CriticGPT rewards, 51.6% ± 14.6 for sparse rewards, and 1.3% ± 1.5 for LIV. The dispersions are sample standard deviations. The average improvement over sparse rewards is useful, but the uncertainty materially changes the takeaway. A standard deviation describes variation among runs; it is not itself a confidence interval on the mean. Nor should twenty evaluation episodes be confused with twenty independent training runs. Evaluating a policy repeatedly reduces uncertainty about that particular policy’s performance. It does not erase variation caused by training another policy. As a diagnostic calculation, treating the two sets of seed-level macro averages as independent gives a standard error for their difference of roughly 13.8 percentage points. That is larger than the ten-point mean difference. This is not a substitute for a paired analysis—cross-method covariance matters—but it shows why I would not budget around a dependable ten-point improvement from these summaries alone. The appropriate conclusion is not that the method has no effect. It is that the experiment supports useful learning behavior more clearly than it establishes a precise, stable aggregate advantage. A regrouping of the published task means is revealing. For the four explicitly identified held-out tasks, I calculate average success of 70% with CriticGPT rewards versus 25% with sparse rewards. Across the other twelve evaluated tasks, the corresponding averages are approximately 58.8% and 60.5%. These are my calculations from the table, not additional experiments. That decomposition makes the result more interesting, not less. The strongest signal is not uniform superiority. It is that transferred preference generation can help on particular tasks where sparse feedback is difficult to learn from. But that interpretation also suggests a mechanism that needs testing. Is the benefit primarily semantic transfer from the critic? Is it simply the availability of an informative dense objective? Could a simpler visual reward trained from comparable source-task supervision deliver the same benefit?. Those questions require matched alternatives, not another comparison against a controller receiving almost no intermediate feedback. There is also a distinction between final performance and efficiency. A final success rate tells us where training ended under the chosen budget. For a practitioner, I would additionally want time to a useful success threshold, area under the learning curve, failure frequency during training, and the total cost of producing the reward system. That last quantity should include the data-generation policies, critic adaptation, preference inference, reward-model fitting, and downstream interaction. A reduction in online interaction can be valuable even if it requires considerable offline work. But the trade-off should be measured rather than hidden behind a single environment-step axis. For diagnosing unstable learning, I would save policy rollouts at multiple checkpoints and rerun three evaluations: human or simulator outcome assessment, CriticGPT comparison, and reward-model ranking. If the first two agree while the student reward diverges, distillation is the likely bottleneck. If the teacher itself misranks newly visited behavior, more student optimization will not fix the underlying supervision. This component-level analysis matters because a policy learning curve is the output of an entire system. A decline cannot, by itself, identify whether the problem lies in perception, reward extrapolation, normalization, exploration, or policy optimization. Are the Baselines Fair—and Are They Current?. The sparse-reward comparison asks a legitimate question: can learned visual feedback make learning easier when completion feedback alone is insufficient?. It does not answer whether the method replaces a well-designed dense reward, nor whether it is superior under equal access to task information. I would include a privileged dense-reward controller as a diagnostic upper reference, while clearly distinguishing that reference from the intended image-based deployment setting. The more decisive ablation would hold the reward-learning pipeline fixed and change only the source of preferences. Use simulator-derived labels, human labels on a smaller subset, a simple trained visual comparator, and the multimodal critic. That would reveal how much performance is gained or lost at the annotation interface. LIV deserves similar care. Its original work learns language-image representations and value-like rewards from text-annotated videos, with both pretrained and target-domain-fine-tuned variants. Its reward-control experiments include model-based planning, rather than only the DrQ-v2 setup used here. Consequently, an almost-failing LIV result in this experiment should trigger an implementation and adaptation audit, not a general verdict on representation-based rewards. I would check image preprocessing, goal specification, reward direction, reward normalization, and sensitivity to scale. Then I would compare pretrained and adapted variants under a documented data budget. Scaling cannot repair an incorrect ordering of states, but it can affect how successfully a particular optimizer learns from an otherwise useful signal. The official LIV repository provides pretrained-model usage, reward-curve generation, and custom-dataset fine-tuning workflows. That makes an adaptation-controlled comparison a concrete experimental option rather than a hypothetical demand. There is a broader fairness issue in comparing a robot-adapted evaluator with an off-the-shelf representation. Such a comparison can establish the usefulness of the complete system. It is weaker evidence that semantic language reasoning, specifically, explains the improvement. For that claim, I would match domain data as closely as possible and vary the architecture. For a practical purchasing decision, I would instead match total engineering and compute effort. Both comparisons are useful; they answer different questions. The publication date also changes the competitive context. SARM, introduced in a 2025 preprint and accepted to ICLR 2026, predicts task stage and within-stage progress using subtask annotations, then uses reward estimates to filter and reweight demonstrations. Its evaluation includes real-world, long-horizon manipulation such as clothing folding. SARM is not a directly interchangeable baseline: its supervision and policy-learning setup differ. Its relevance is that reward quality can be operationalized around stages and useful training data, rather than around a language model’s overall preference between complete attempts. Robo-Dopamine 2.0, an August 2026 preprint, explicitly conditions pairwise progress prediction on history and trains for distinctions among valid progress, failure, and recovery. It includes both simulated policy learning and real-world insertion experiments. Again, different tasks and data prevent a numerical leaderboard comparison. But these directions sharpen the standard for a contemporary generalization claim. A useful robotic evaluator should eventually distinguish a harmless visual change from a failed manipulation, and a recovery from a superficially similar retreat. CriticGPT’s early contribution remains relevant without claiming that its journal evaluation settles that broader competition. What Transfers—and What Does Not?. Factor World is built on Meta-World to vary environmental factors systematically. Its original benchmark was designed to separate sources of visual-manipulation generalization difficulty, rather than treat every distribution shift as one undifferentiated test. CriticGPT’s camera-shift accuracy falls from 95.9% to 86.6%. Each condition uses a fixed 300-pair set evaluated through five stochastic inference runs. That is useful evidence about viewpoint sensitivity. The statistical unit is important, however: repeated decoding on a fixed dataset tells us something about inference variability, not about variation across independently collected environments or independently trained critics. For deployment, I would want both. The same model should be evaluated across fresh scene configurations, and multiple trained models should be evaluated if training reproducibility is part of the claim. Single-factor tests are also best viewed as diagnostics. They identify which changes deserve attention. They do not establish performance when changes interact—for example, a shifted camera making a critical contact harder to see under different lighting. A practical follow-up would start with the weakest factor and compose it with others. That is more informative than reporting an average across many easy perturbations that masks one severe failure mode. Task transfer needs equally precise language. Reusing a critic to label a new task is not the same as reusing one reward function, and neither is the same as executing a new task without policy training. A deployment plan must specify which components remain fixed and which still require target-task data and optimization. My preferred interpretation is therefore narrow: this is evidence for transferring an annotation capability within a related simulation family. It should motivate, rather than replace, experiments on new embodiments, different contact dynamics, and physical robots. The sim-to-real question is particularly demanding for a visual critic. Consider an insertion that looks aligned from the external camera but is mechanically jammed, or a grasp that appears secure until the object slips. More visual fluency does not automatically reveal information absent from the observation. That suggests two possible responses. Improve observability through additional views or sensing, or restrict the evaluator’s contract to outcomes that are genuinely visible. Simply increasing model capacity is not a substitute for deciding which of those problems you have. Can a Competent Team Reproduce It?. The paper provides an implementation checklist, but its availability statement still describes a future public release. As of October 14, 2026, I could not verify a public CriticGPT code-and-checkpoint release through the paper and the lead author’s publication entry; that entry links the paper rather than a dedicated implementation. This is a verification limitation, not proof that no repository exists elsewhere. I would distinguish three levels of reproducibility here. A competent team can understand and reimplement the high-level recipe. Reproducing the same learning behavior requires more: data construction rules, trajectory provenance, task splits, input formatting, and reward-model training details. Reproducing the numerical results requires the actual artifacts and evaluation configuration. The most consequential questions I would send the authors concern interfaces rather than optimizer trivia. Which exact language-model checkpoint was used? How are frame features packed or compressed into the language context? How are ties handled during data construction and training? How many unique trajectories underlie the pairs? What target-task preference budget is required?. That last distinction matters because many pairs can be constructed from a much smaller set of underlying episodes. Pair count alone does not tell us how much behavioral diversity the evaluator has seen. I would also request the full simulator-to-text template. It is part of the effective training objective. Small changes in whether success dominates distance, or in how ambiguous attempts are treated, could alter the learned preferences without changing the neural architecture at all. For compute planning, parameter-efficient adaptation should not be mistaken for cost-free inference. Before allocating a reproduction effort, I would measure video-encoding time, language-generation time, memory use, and annotation throughput on the intended hardware. The release I would find most useful would include a small deterministic end-to-end reproduction: fixed trajectory IDs, fixed preference outputs, one reward-model fit, and one policy run. That would establish a working reference before asking another team to reproduce a broad multi-task result. What This Means for Practitioners. I would consider CriticGPT as a reward-learning design pattern and research starting point, not as a validated general-purpose judge ready to supervise arbitrary manipulation. The part worth borrowing is the separation of responsibilities. Let a comparatively expensive model evaluate rich behavioral evidence. Convert that feedback into an interface a conventional learning system can use. Keep the teacher, reward model, and controller independently inspectable. My first experiment would be offline. Assemble an adjudicated evaluation set that includes obvious successes, obvious failures, near-successes, recovery attempts, occlusion, and cases where final appearance is misleading. Preserve an explicit tie or uncertain category rather than forcing every ambiguous pair into a confident ranking. I would then test consistency under swapping the order of the two attempts. A reliable preference interface should not change its underlying decision merely because the clips exchange positions. I would separately test frame-order sensitivity to determine whether the evaluator uses sequence information in the way the task requires. Next, audit the distilled reward—not just the language model. Does it rank unseen attempts correctly? Does it distinguish completion from lingering near completion? Can a sequence receive a high total while failing the actual task?. This is where a polished natural-language explanation can become a distraction. The controller optimizes the numerical objective, not the explanation. The reward network’s behavior on policy-generated observations is the consequential artifact. For an initial learning trial, I would keep an independently defined success measurement and compare against at least one inexpensive reward-learning alternative under the same target-task data budget. The aim is to discover whether the multimodal teacher buys useful transfer or better supervision, rather than merely adding another model to the pipeline. Finally, measure the economics honestly. The value proposition is not simply fewer human preference clicks. It is the total cost of acquiring behavioral coverage, generating and validating labels, fitting rewards, debugging failures, and collecting online robot experience. What the evidence supports is a plausible bridge from video comparison to policy learning. What remains uncertain is the reliability of that bridge under optimization, new physical conditions, and realistic deployment costs. The existing experiments cannot establish that a convincing visual critic is a universal reward oracle. The practical lesson is to audit both sides of the interface: the large model as an annotator, and the small reward model as an objective a policy will actively optimize. CriticGPT makes that interface worth investigating. It does not make either audit optional.