← 首页|学术|On-policy Distillation with Verifiable Reward
cs.LG, cs.AI · 2608.24696 · 2026-08-25

On-policy Distillation with Verifiable Reward

Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, et al.
RLVROn-Policy DistillationPost-Training
💬 不引入任何新超参数,把 RLVR 的任务级正确性信号和 OPD 的密集 token 级指导无缝融合:用 ReLU 门控让正确轨迹拿到非负奖励、错误轨迹拿到非正奖励,把 OPD 直接转化为一个规范的 RLVR 方法。

🎯 背景

RLVR(可验证奖励强化学习)和 OPD(在线策略蒸馏)是当前 LLM 后训练两大广泛采用的范式。RLVR 的任务级反馈稀疏,OPD 提供密集的 token 级指导但忽略轨迹正确性,性能因此被限制在教师模型的水平之内。把两者结合是一个有前景的方向:OPD 提供密集监督信号,RLVR 提供任务级正确性;但现有的融合方法大多依赖加权组合或启发式切换,引入了额外的超参数和权衡。

🔬 方法

论文提出 OPDVR(On-policy Distillation with Verifiable Reward):先基于轨迹正确性重新表述采样 token OPD 的隐式奖励,再应用一个 ReLU 门控机制,确保正确轨迹获得非负奖励、错误轨迹获得非正奖励——从而把蒸馏信号与任务成功对齐,同时保留教师模型的分布指导。这一改动把采样 token OPD 转化为一个规范的 RLVR 方法,因而可以与任意策略梯度算法(如 GRPO)直接组合,且不引入任何新超参数。

📊 结果

在 6 个推理基准上,OPDVR 持续优于标准 OPD。

原文摘要

▶ Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at this https URL .
来源: arXiv:2608.24696 · 精读由高松灯生成,基于摘要与 arXiv 页面信息