VINE

VINE: Taming Generative Control Policies for Reinforcement Learning

Enabling stable backpropagation-through-time fine-tuning of flow-matching policies

1 AgiBot   2 The Hong Kong University of Science and Technology   3 Peking University

šŸ“„ Paper šŸ’» Code (Coming Soon)

Abstract

Flow-matching policies have emerged as an effective policy parameterization for robot learning. They iteratively generate actions from noise, enabling highly expressive modeling of complex and multimodal action distributions. However, prior works observed that scaling these policies with value-gradient reinforcement learning (RL) often leads to training instability. Existing methods attribute this instability to iterative generation and therefore avoid end-to-end value-gradient optimization by sacrificing iterative generation, high expressiveness, or value-gradient optimization. Contrary to prior belief, we show the instability does not stem from iterative generation itself, but from the vanilla sampling strategy originally designed for behavior cloning, which becomes brittle under value-gradient RL. Motivated by this insight, we propose VINE, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies. Instead of following a single flow trajectory, VINE reconstructs a new interpolation state at every denoising step, creating a stable differentiable path for value-gradient propagation while remaining compatible with the original flow-matching denoising process. As a result, VINE preserves the expressiveness and iterative generation of flow-matching without sacrificing end-to-end value-gradient optimization. Despite performing end-to-end backpropagation through all ten denoising steps, VINE achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation tasks.

VINE overview: stable end-to-end value-gradient fine-tuning for flow-matching policies.

VINE reconstructs a new interpolation state at each denoising step, enabling stable end-to-end value-gradient optimization while preserving iterative flow-matching generation.

Algorithm

Click to see the full algorithm

Algorithm 1 VINE: Actor-Critic Training

  1. function Generate(s)
  2. a0 āˆ¼ N(0, Id)
  3. for k = 0, ā€¦, K āˆ’ 1 doā–¹ Iterative generation
  4. tk ā† k/K
  5. xk ā† xk + (1/K) vĪø(xk, tk; s)ā–¹ Replace Euler Method
  6. zk āˆ¼ N(0, Id)
  7. xk ā† tk ak + (1 āˆ’ tk) zk
  8. ak+1 ā† xk + (1 āˆ’ tk) vĪø(xk, tk; s)
  9. end for
  10. return aK
  11. end function
  12. while not converged do
  13. Collect transitions with Generate(s) and add to Dā–¹ Optionally for online RL
  14. Sample batch {(s, a, r, sā€²)} āˆ¼ D
  15. aā€² ā† Generate(sā€²)
  16. Update Ļ† to minimize E[(QĻ†(s, a) āˆ’ r āˆ’ Ī³ QĻ†Ģ„(sā€², aā€²))2]ā–¹ Train critic QĻ†
  17. aK ā† Generate(s)
  18. Update Īø to minimize āˆ’QĻ†(s, aK) + Ī±ā€–aK āˆ’ aā€–2ā–¹ Train velocity vĪø via BPTT
  19. end while
  20. return policy Ļ€Īø(s) ā‰” Generate(s)

Experiments

Offline RL on OGBench

We first test whether VINE can optimize expressive generative policies in offline RL. The benchmark covers 40 tasks from eight OGBench domains, including long-horizon navigation and sparse-reward manipulation. Each task is trained with 12 seeds and reported as mean performance with 95% confidence intervals.

Offline RL results on 40 OGBench tasks
Click to expand offline results across 8 domains (40 tasks)

Offline results, 8 domains. Per-task success rates with 95% bootstrap confidence intervals (12 seeds per task). Each domain reports five tasks plus a pooled aggregate.

Offline RL results across 8 OGBench domains and 40 tasks

Toy Simulation: BC vs. RL

On a 2D multimodal toy task, we compare behavior cloning (BC) with TD3+BC fine-tuning for diffusion (DDPM), flow matching with Euler sampling (FM), and VINE. Each animation shows sampled actions and critic values during inference.

BC vs RL comparison for DDPM, flow matching, and VINE on a toy multimodal task

DDPM

BC. Diffusion policy after behavior cloning.
RL (TD3+BC). Diffusion policy after value-gradient fine-tuning.

Flow Matching (Euler)

BC. Flow-matching policy with Euler sampling after behavior cloning.
RL (TD3+BC). Flow-matching policy with Euler sampling after fine-tuning.

VINE

BC. Flow-matching policy with VINE sampling after behavior cloning.
RL (TD3+BC). VINE policy after stable value-gradient fine-tuning.

Online Real-World Socket Insertion

We further evaluate VINE on a real-world plug insertion task. Starting from scratch, the robot learns to pick up a plug and insert it into a fixed socket under tight contact tolerances. VINE reaches 20/20 success after 20 minutes of fine-tuning, with 19.2% human-intervention steps over the collected data.

Real-World Rollout Comparison

Scroll horizontally through each row to compare successful VINE rollouts with representative baseline failure cases on the socket insertion task.

VINE Success Cases

Scroll to view more success clips ā†’

VINE success case. The policy aligns the plug and completes the insertion reliably.
VINE success case. Another successful rollout with stable contact-rich insertion.

Baseline Failure Cases

Scroll to view more failure clips ā†’

SAC-Flow failure case. The policy approaches the socket but cannot complete the insertion.
RLT failure case. The rollout stalls before achieving a stable insertion.
EXPO failure case. The policy misses the precise alignment needed for contact-rich insertion.
DSRL failure case. The robot fails to recover from misalignment during insertion.

Learning Progression from Scratch

This 25-minute run shows VINE can train a real-world manipulation policy entirely from scratch. Without any behavior cloning warm-up or demonstrations, both the actor and critic heads are randomly initialized and optimized through human-in-the-loop reinforcement learning. The policy uses only a lightweight 3M-parameter network with a ResNet-10 visual encoder. Starting from random exploration, it discovers the socket within a few minutes, quickly transforms isolated successes into consistent behavior, and achieves 20 consecutive successful insertions after only 20 minutes of real-world interaction.

From-scratch online fine-tuning on real-world socket insertion with VINE. The policy reaches reliable performance within 25 minutes.
Time Observed behavior
0:00 Random motion; no meaningful task progress yet.
2:00 The arm finds the socket region, but insertion remains 0/20.
4:20 First autonomous success, still fragile and hard to repeat.
5:23 Small corrections appear; short 3ā€“4 success streaks emerge.
8:00 Human intervention drops to light corrections.
13:00 Faster, bolder motions briefly trade off precision.
20:00 Stable speed and accuracy; 20 consecutive successful insertions.
Plug Insertion Task BC init. DSRL RLT EXPO SAC-Flow Hil-SERL VINE
Success rate (ā†‘)
Time (ā†“)		 10/20
--		 17/20
50 min		 17/20
50 min		 19/20
40 min		 12/20
20 min		 16/20
20 min		 20/20
20 min
Human-intervention (ā†“) -- -- 29.7% 56.3% 74.6% 55.9% 19.2%

BC policy is pi05 trained from 15 human demonstrations. Human-intervention ratio is computed as intervention steps / total rollout steps. DSRL, RLT, and EXPO use a BC policy for base, while VINE, SAC-Flow, and Hil-SERL learn directly from scratch without BC pretraining.

More Real-World Experiments

Additional from-scratch real-world fine-tuning runs with VINE on contact-rich manipulation tasks.

Screw insertion. VINE is trained from scratch, converges within 5 minutes, and then completes 20 consecutive successful trials.
Plug insertion. VINE reaches 20 consecutive successful insertions within 15 minutes of from-scratch online fine-tuning.

Citation

@misc{yang2026vinetaminggenerativecontrol,
      title={VINE: Taming Generative Control Policies for Reinforcement Learning}, 
      author={Rushuai Yang and Zhuo Han and Houlin Li and Hecheng Wang and Zhichao Wu and Rui Zhang and Zhaowei Zhang and Zihong Chen and Xiaohan Yan and Chiming Liu and Yi Chen and Wei Shan and Maoqing Yao},
      year={2026},
      eprint={2607.10369},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.10369}, 
}