RARM

Confidence-Gated Progress Reward Modeling for RL in Manipulation

Pengzhi Yang^†*, Xinyu Wang^‡*, Pengyu Jing^†*, Kehan Wen^†*, Yiduo Qu^†§, Zhenhao Huang^†, Minghao Fu^¶, Xin Liu^†, Yaheng Shen^†, Fan Shi^† ^† NUS Human-Centered Robotic Lab · ^‡ Booking.com · ^§ University of Cambridge · ^¶ School of Artificial Intelligence, Nanjing University ^* Equal contribution · Corresponding author: Fan Shi, fan.shi@nus.edu.sg

Paper Twitter Code Coming soon

Overview

Abstract

Reinforcement learning for robot manipulation is often bottlenecked by reward design, especially in long-horizon tasks: sparse success rewards provide weak supervision, while hand-crafted dense rewards are tedious to design and generalize poorly across tasks. Progress-based reward models offer a promising alternative by estimating how far an observation has advanced toward task completion, but existing approaches often require task-specific demonstrations or progress labels, and can assign high rewards to visually plausible but physically incorrect states. We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward. RARM is trained once on general-purpose videos with a contrastive temporal objective, requiring no robot-specific data, task-specific reward labels, or per-task reward engineering. At deployment, RARM matches rollout clips to reference clips and rewards only confident forward progress, suppressing uncertain matches that may otherwise produce false-positive rewards. Across 9 simulated manipulation tasks from LIBERO and MetaWorld and 4 real-world tasks, RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding, where unreliable progress estimates are especially harmful.

Our Contributions

Data-light, reference-anchored progress reward

We introduce RARM, which turns a single successful demonstration into a dense progress reward with no task-specific reward supervision—achieving data-light, confidence-aware progress estimation where prior work meets only one of these criteria.

Confidence gating against reward hacking

RARM rewards confident forward progress and explicitly filters out low-confidence matches, suppressing the false-positive rewards that drive reward hacking.

Best overall success with low overhead

Across 9 simulated tasks (LIBERO, MetaWorld) and 4 real-world tasks, RARM achieves the best overall success rates with low-overhead reward queries, with especially large gains on long-horizon settings such as cloth folding.

Method

Method Overview. Reward Model Training: As in (c), we sample an anchor clip from an unlabeled video, positives from the same temporal region, and negatives from three complementary sources. These pairs are scored by the comparator in (a), trained with a soft-nearest-neighbours loss. RL Training: Given a reference and a rollout video in (b), each rollout clip is compared with all reference clips to form the matrix in (d). Rather than using these similarities directly as rewards, RARM uses them to estimate where the rollout lies along the reference, reading progress from the best-matched reference clips. To avoid false-positive progress, this estimate is passed through a confidence gate in (e): uncertain matches are suppressed, so the policy is not rewarded when progress localization is unreliable. The policy is updated with the resulting reward as new rollouts are collected iteratively. The confidence threshold is computed once from self-comparisons within the reference demonstration, which sets how strong a valid match should be.

Simulation Experiments

Qualitative Comparisons

RARM's cumulative reward along a successful vs. a failed rollout of the same task. This is the running total of RARM's reward, not a raw progress estimate: RARM matches each clip against references with a confidence gate, and adds +1 whenever a confident match beats the highest progress reached so far. Hover the plot to scrub through the rollout — the dots track each curve and the frames update to the matching step.

Qualitative comparison chart unavailable

Quantitative Experiments

Success rate vs. environment steps on MetaWorld and LIBERO-10 tasks (mean ± std over seeds) with Drq-v2 from scratch. Hover the legend to highlight a method; click to toggle it.

Success-rate curves for RARM (ours) and baselines across MetaWorld and LIBERO-10 tasks.

Ablation

Each variant disables one part of RARM. The full model (blue) reaches a high success rate the fastest and holds it. w/o Cross-Attn replaces the learned cross-attention comparator with a simpler distance-based comparison, weakening rollout–reference matching and degrading progress localization; w/o Temporal loses motion cues and rewards static frames that merely look like success. The confidence gate is the key factor: without it (w/o Gate) noisy frame matches inject spurious reward and convergence slows, while an over-strict gate (Strict Gate) rejects too many frames and starves the policy — both plateau well below RARM.

Real-World Experiments

DSRL with RARM

Drawer Opening

Pick and Place Eraser

Handover

Clothes Folding

Cumulative Predicted Reward on Clothes Folding

Cumulative predicted reward on paired success and failure cloth-folding rollouts.

Cumulative predicted reward on a paired success / failure cloth-folding rollout, normalized per model by its own success-rollout final reward. Reward is accrued only when newly estimated progress exceeds the previous highest estimate, so each curve is monotonically non-decreasing — transient mis-predictions cannot inflate it. Ours (red) tracks the linear oracle on success and saturates at just 0.50 on failure, while every baseline assigns more than 80% of its success reward to the failed rollout (GVL 0.97, Robometer 0.88, RoboDopamine 0.84, VIP 0.82) and cannot separate the two. Removing the per-clip similarity threshold (Ours w/o threshold) accepts every nearest-demo match as confident, and the failure rollout then earns more reward than success (ratio ≈ 1.89) — confirming that confidence-gated progress estimation is what makes success and failure distinguishable.

Real-World Success Rates

Success rates over 10 evaluation rollouts before and after 30/60/90 DSRL training episodes. w/o DSRL: base VLA policy. BinR: DSRL with sparse binary success reward. Ours: DSRL with the proposed RARM reward.

Task	w/o DSRL	30 rollouts		60 rollouts		90 rollouts
Task	w/o DSRL	BinR	Ours	BinR	Ours	BinR	Ours
Drawer open	2/10	4/10	6/10	6/10	7/10	8/10	10/10
Pick eraser	2/10	3/10	6/10	5/10	8/10	8/10	9/10
Hand over	2/10	2/10	5/10	4/10	6/10	5/10	6/10
Clothes folding	1/10	0/10	1/10	1/10	3/10	2/10	8/10

Robustness for RARM

RARM produces stable progress estimates under diverse visual perturbations. Starting from a successful bimanual clothes-folding rollout, we apply four augmentations targeting distinct real-world modes and re-estimate progress against an unaltered reference. In all cases the estimated progress closely tracks the ideal linear trend.

Reference Demonstration — Task 4: Bimanual Clothes Folding

Bimanual first fold — Bimanual
first fold

Cooperative 45° rotation — Cooperative
45° rotation

Left pinches near side — Left pinches
near side

Augment 1 — Glareblinding white spots

Progress estimate under glare augmentation

Augment 2 — Color Shiftgreen filter

Progress estimate under color shift augmentation

Augment 3 — Occlusionblack box overlay

Progress estimate under occlusion augmentation

Augment 4 — Viewpointperspective pitch & yaw

Progress estimate under viewpoint augmentation

Computational Cost

Inference Speed and GPU Cost

Per-frame inference cost and peak GPU memory across reward model baselines, measured over 125 frames (one rollout trajectory). RARM matches the fastest vision-encoder baselines while using far less memory than VLM-based methods. These comparisons are all conducted on desktop with one 5090.

Baseline	ms / frame	Mem (MB)
GVL	<50	≈13 200
Robometer	<100	≈13 200
RoboDopamine	<300	≈19 000
VIP	<10	≈4 200
RoboCLIP	<10	≈5 400
TemporalOT	<10	≈6 600
RARM (Ours)	<10	≈5 100

BibTeX

@misc{yang2026rarm,
  title  = {RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation},
  author = {Yang, Pengzhi and Wang, Xinyu and Jing, Pengyu and Wen, Kehan and Qu, Yiduo and Huang, Zhenhao and Fu, Minghao and Liu, Xin and Shen, Yaheng and Shi, Fan},
  year   = {2026}
}