VINE

VINE: Taming Generative Control Policies for Reinforcement Learning

Stable value-gradient reinforcement learning through the full iterative sampling chain

1 AgiBot   2 The Hong Kong University of Science and Technology   3 Peking University

šŸ“„ Paper šŸ’» Code (Coming Soon)

Abstract

Generative control policies use diffusion and flow matching to iteratively transform noise into actions, enabling complex action distributions. Direct reinforcement learning (RL) optimization can nevertheless be unstable when critic value gradients propagate through the full generation chain. We introduce VINE (Value-gradient Iterative Noise Exploration), a flow-matching sampling strategy that reconstructs the generation path at each iteration with fresh Gaussian noise. VINE changes only the vanilla flow-matching sampler while retaining iterative generation and end-to-end value-gradient optimization through the entire sampling chain. Across OGBench, online adaptation of the pretrained 3B-parameter Ļ€0.5 VLA on LIBERO, and real-world plug insertion, VINE supports stable value-gradient RL in the evaluated settings.

VINE overview: stable end-to-end value-gradient fine-tuning for flow-matching policies.

VINE reconstructs intermediate sampling states with fresh Gaussian noise, while propagating value gradients through the full iterative generation chain.

Algorithm

Click to see the full algorithm

VINE: Actor-Critic Training. Here, s and a are the environment state and action, xk is the intermediate sampling state, and K is the number of velocity-network evaluations. The hat denotes an action-endpoint estimate.

  1. function Generate(s)
  2. a0 āˆ¼ N(0, Id)
  3. for k = 0, ā€¦, K āˆ’ 1 doā–¹ Iterative generation
  4. tk ā† k/K
  5. xk+1 ā† xk + 1/K vĪø(xk, tk; s)ā–¹ Euler update
  6. zk āˆ¼ N(0, Id)ā–¹ VINE update
  7. xk ā† tk ak + (1 āˆ’ tk) zk
  8. ak+1 ā† xk + (1 āˆ’ tk) vĪø(xk, tk; s)
  9. end for
  10. return aK
  11. end function
  12. while not converged do
  13. Collect transitions with Generate(s) and add to Dā–¹ Optionally for online RL
  14. Sample batch {(s, a, r, sā€²)} āˆ¼ D
  15. aā€² ā† Generate(sā€²)
  16. Update Ļ† to minimize E[(QĻ†(s, a) āˆ’ r āˆ’ Ī³ QĻ†Ģ„(sā€², aā€²))2]ā–¹ Train critic QĻ†
  17. aK ā† Generate(s)
  18. Update Īø to minimize āˆ’QĻ†(s, aK) + Ī±ā€–aK āˆ’ aā€–2ā–¹ Train velocity vĪø via BPTT
  19. end while
  20. return policy Ļ€Īø(s) ā‰” Generate(s)

Terminal critic targets retain the observed reward and omit the next-state value term. During actor updates, critic parameters are fixed while their action gradients pass through every endpoint prediction and reconstruction; no intermediate state is detached.

Toy Simulation: BC vs. RL

We compare DDPM, Euler flow matching, and VINE on a two-dimensional five-mode action distribution with different rewards. The first row shows behavior-cloned policies; the second shows offline RL fine-tuning with a learned critic. In the paper, DDPM and Euler flow matching eventually produce increasingly large off-target actions, whereas VINE retains concentrated endpoints while shifting toward the highest-reward mode. The appendix additionally reports direct optimization of a known reward without a learned critic.

Training DDPM Flow Matching (Euler) VINE
BC
Diffusion policy after behavior cloning.
Flow-matching policy with Euler sampling after behavior cloning.
Flow-matching policy with VINE sampling after behavior cloning.
RL
Diffusion policy after value-gradient fine-tuning.
Flow-matching policy with Euler sampling after fine-tuning.
VINE policy after value-gradient fine-tuning.

Experiments

Real-World Experiments

Real-world manipulation with VINE, from precise insertion to dual-arm and tabletop tasks.

Screw insertionTraining time: VINE ā€” 5 min vs. HIL-SERL ā€” 1 h.
Plug insertionTraining time: VINE ā€” 20 min vs. HIL-SERL ā€” 1 h 40 min.
Dual-arm panel pressingTraining time: VINE ā€” 1 h 1 min vs. HIL-SERL ā€” 2 h 2 min.
Material placementOnline RL: VINE-Ļ€0.5 ā€” 5 h; success rate: 30% ā†’ 70%.

Online Real-World Socket Insertion

The robot must pick up a plug from the table and insert it into a fixed socket. All methods use images and proprioceptive observations to predict low-level actions or action chunks; human operators may intervene and provide corrective actions during online training. After 20 minutes of online RL, VINE succeeds in 20/20 evaluation trials and has a 19.2% human-intervention ratio.

Real-World Rollout Comparison

Selected VINE successes and baseline failures on socket insertion. These clips illustrate behavior; the table below reports success over 20 evaluation trials per method.

VINE Success Cases

VINE success case. The policy aligns the plug and completes the insertion.
VINE success case. Another successful rollout with stable contact-rich insertion.

Baseline Failure Cases

SAC-Flow failure case. The policy approaches the socket but cannot complete the insertion.
RLT failure case. The rollout stalls before achieving a stable insertion.
EXPO failure case. The policy misses the precise alignment needed for contact-rich insertion.
DSRL failure case. The robot fails to recover from misalignment during insertion.

Online Learning with Human Feedback

This recording shows VINE training on real-world socket insertion. A human operator can intervene during autonomous execution and provide corrective actions. The evaluation below reports task success, online training time, and intervention frequency separately.

Online RL training with VINE on real-world socket insertion.
Protocol Setting
Task Pick up a plug and insert it into a fixed socket.
Observations Images and proprioceptive states.
Actions End-effector control.
Human feedback Corrective actions during online rollouts.
Evaluation 20 trials per method after the listed training budget.
Method Success (ā†‘) Online RL time Human intervention (ā†“)
BC init. 10/20 -- --
DSRL 17/20 50 min --
RLT 17/20 50 min 29.7%
EXPO 19/20 50 min 56.3%
SAC-Flow 12/20 20 min 74.6%
HIL-SERL 16/20 20 min 55.9%
VINE 20/20 20 min 19.2%

Success is measured over 20 evaluation trials. Human-intervention ratio is the fraction of online rollout steps under human control. Training times are the reported evaluation budgets, not times to a common convergence threshold. A dash denotes an unreported or inapplicable entry.

Multi-Task Online RL with a Pretrained VLA

We adapt the 3B-parameter Ļ€0.5 VLA on LIBERO-10, LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal, each with ten tabletop manipulation tasks. All variants share the same few-shot SFT initialization and differ only in the sampler: deterministic ODE, stochastic SDE, or VINE. In the evaluated setting, VINE yields more consistent policy improvement, while the ODE and SDE variants are unstable during multi-task fine-tuning. Each curve reports success averaged over the ten tasks in its suite.

Online RL adaptation of Ļ€0.5 on LIBERO-10, Spatial, Object, and Goal, comparing ODE, SDE, and VINE sampling
Online RL adaptation on LIBERO. Success is averaged over the ten tasks in each suite. Faint lines show the original observations; bold lines show five-point trailing means. Click the figure to open the original PDF.

In a separate frozen-checkpoint evaluation on LIBERO-10, Euler and VINE achieve 44.34% and 45.26% success, respectively, across ten evaluation seeds with 500 episodes per seed and sampler. These similar observed rates provide empirical evidence that the pretrained flow-matching checkpoint can be directly reused with VINE.

Offline RL on OGBench

We evaluate 50 state-based tasks across ten OGBench domains, spanning long-horizon navigation and sparse-reward manipulation. Each task is trained for 1M gradient updates with 12 random seeds; results report mean success with 95% confidence intervals.

Aggregate success on 50 OGBench tasks with 95% confidence intervals
Click to expand offline results across 10 domains (50 tasks)

Offline results, 10 domains. Domain-level and overall mean success (%) with 95% confidence intervals in brackets, as reported in the paper. Each domain contains five tasks, with 12 training seeds per task. BPTT-free methods avoid propagating value gradients through the full sampling chain; Value-Gradient BPTT methods retain this optimization path.

Task category Gaussian BPTT-free Value-Gradient BPTT
ReBRAC FQL FAWAC IFQL QAM DAC QSM CGQL CGQL-M CGQL-L DSRL FEdit FBRAC BAM VINE
antmaze-large 94 [94, 95] 76 [72, 79] 17 [15, 19] 36 [32, 39] 81 [78, 84] 88 [86, 90] 91 [89, 93] 76 [73, 80] 71 [68, 73] 65 [62, 67] 61 [56, 66] 58 [53, 62] 2 [1, 4] 84 [82, 85] 99 [98, 100]
antmaze-giant 57 [53, 60] 0 [0, 0] 0 [0, 0] 1 [0, 2] 18 [14, 22] 16 [11, 21] 15 [11, 17] 0 [0, 2] 4 [1, 8] 3 [1, 6] 3 [1, 4] 2 [1, 3] 0 [0, 0] 1 [0, 2] 76 [68, 83]
humanoidmaze-medium 69 [65, 74] 68 [63, 73] 24 [22, 26] 86 [85, 87] 67 [64, 69] 83 [81, 85] 83 [80, 86] 60 [57, 62] 42 [40, 43] 62 [57, 67] 53 [48, 57] 22 [20, 23] 39 [37, 41] 60 [58, 62] 87 [79, 93]
humanoidmaze-large 17 [15, 20] 9 [7, 11] 0 [0, 0] 24 [21, 27] 11 [9, 14] 0 [0, 0] 10 [9, 11] 5 [4, 5] 6 [3, 8] 6 [5, 8] 3 [2, 5] 3 [2, 3] 0 [0, 0] 5 [4, 8] 46 [36, 56]
scene-sparse 65 [61, 69] 78 [77, 80] 38 [35, 41] 84 [83, 85] 97 [96, 98] 68 [65, 70] 86 [84, 87] 38 [36, 40] 74 [72, 76] 88 [85, 91] 99 [99, 100] 62 [60, 65] 50 [43, 57] 98 [97, 99] 73 [60, 85]
puzzle-3x3-sparse 79 [73, 84] 70 [60, 78] 3 [2, 3] 100 [100, 100] 100 [99, 100] 68 [62, 75] 53 [49, 57] 48 [39, 55] 100 [100, 100] 90 [83, 96] 87 [82, 92] 99 [98, 100] 0 [0, 1] 56 [48, 64] 95 [93, 97]
puzzle-4x4-100M-sparse 0 [0, 0] 5 [3, 7] 0 [0, 0] 0 [0, 0] 0 [0, 0] 0 [0, 0] 0 [0, 0] 24 [16, 33] 0 [0, 0] 0 [0, 0] 0 [0, 0] 34 [30, 37] 15 [12, 19] 0 [0, 0] 28 [20, 35]
cube-double 9 [8, 10] 46 [43, 49] 2 [2, 2] 11 [10, 12] 64 [62, 66] 35 [33, 36] 56 [53, 58] 38 [36, 41] 41 [39, 43] 45 [43, 47] 74 [72, 76] 40 [37, 43] 0 [0, 0] 47 [44, 50] 35 [29, 41]
cube-triple 1 [0, 1] 3 [2, 4] 0 [0, 0] 0 [0, 0] 3 [3, 4] 5 [3, 6] 3 [3, 4] 8 [7, 9] 8 [7, 9] 8 [7, 9] 1 [1, 2] 2 [2, 3] 0 [0, 1] 3 [2, 5] 6 [4, 7]
cube-quadruple-100M 9 [6, 11] 2 [1, 4] 0 [0, 0] 2 [1, 3] 3 [2, 4] 3 [1, 5] 19 [19, 20] 0 [0, 0] 1 [0, 1] 0 [0, 1] 2 [2, 3] 5 [3, 7] 0 [0, 0] 0 [0, 0] 47 [38, 56]
all (50 tasks) 40 [39, 41] 36 [34, 37] 8 [8, 9] 34 [34, 35] 44 [44, 45] 36 [35, 38] 42 [41, 42] 30 [28, 31] 35 [34, 35] 37 [36, 38] 38 [38, 39] 33 [32, 33] 11 [10, 12] 35 [34, 36] 59 [56, 62]

Citation

@misc{yang2026vinetaminggenerativecontrol,
      title={VINE: Taming Generative Control Policies for Reinforcement Learning}, 
      author={Rushuai Yang and Zhuo Han and Houlin Li and Rui Zhang and Hecheng Wang and Zhichao Wu and Zhaowei Zhang and Zihong Chen and Xiaohan Yan and Chuankang Li and Guanghui Ren and Chiming Liu and Yi Chen and Wei Shan and Maoqing Yao},
      year={2026},
      eprint={2607.10369},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.10369}, 
}