← 首页|学术|I-SDPO: Instance-Level Adaptive Self-Distillation
cs.LG · cs.CL · 2608.12957 · 2026/08/13

I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

Zhang, Yubo, Ma, Xinhong, Tan, Zezhong, Dong, Ziqiang
TL;DR:按实例路由:全错的 rollout 组用特权自蒸馏、有成功样本的组继续 GRPO——只在组相对奖励无信息时才用模仿。SciknowEval 四个领域全部最佳,mean@16 从 56.67% 提到 70.31%。

🎯 问题

GRPO 靠组内奖励差学习,但整组都错时无有效相对信号;全程用特权自蒸馏则教师是偏置、低方差的奖励替代,策略能产出成功轨迹后持续模仿反而对抗奖励改进。

🔬 方法

I-SDPO 把教师依赖视为能力相关:每个输入实例一次路由决策、共享给整组——全错组用特权自蒸馏目标,有成功组保持 GRPO。局部分析刻画教师与奖励何时对齐,非零偏置蒸馏权重造成优化偏置地板;路由规则随成功率自动降低蒸馏率,无需手工调度。
章节结构(全文标题提取):
1 Introduction
2 Related Work
3 Method
4 Experiments
5 Discussion
6 Limitations
7 Conclusion
3.1 Preliminaries · 3.2 The Degenerate Gradient Problem · 3.3 Why Teacher Trust Should Depend on Capability · 3.4 I-SDPO: Instance-Level Routing · 3.5 Privileged Teacher for Self-Distillation · 3.6 Entropy-Aware Dynamic Weighting · 3.7 Self-Annealing Property · 4.1 Experimental Setup · 4.2 Main Results · 4.3 Training Dynamics Analysis · 4.4 Capability-Dependent Teacher Trust

📊 结果

SciKnowEval 四个科学领域全部最佳,mean@16 准确率 GRPO 56.67% → 70.31%,最大单领域 +18.24。

💡 与研究方向关联

「何时该模仿教师、何时该靠奖励」的自适应路由,正是 agentic RL 里 RLVR 与蒸馏融合的核心开关,与 CrEST/Temporal GRPO 组成今日信用分配三连。

📝 原文摘要

▶ 原文摘要 Abstract
Group Relative Policy Optimization (GRPO) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect. Privileged self-distillation can fill this gap with dense token supervision, yet applying it throughout training creates a different failure mode: the teacher is a biased, low-variance surrogate for the reward objective, so persistent imitation can oppose reward-improving updates after the policy becomes capable of producing successful trajectories. We introduce I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), which treats teacher reliance as capability-dependent. I-SDPO makes one routing decision per input instance and shares it across that instance's rollout group: all-incorrect groups use a privileged self-distillation objective, whereas any-success groups remain intact for GRPO. This design uses imitation only where group-relative rewards are uninformative. A local analysis characterizes when teacher and reward directions align and shows that a non-vanishing biased distillation weight induces an optimization bias floor. The routing rule automatically reduces the expected distillation rate as success probability rises, withdrawing teacher influence without a hand-designed schedule. On SciKnowEval, I-SDPO obtains the best result in all four scientific domains and improves average mean@16 accuracy from 56.67% with GRPO to 70.31%, with a maximum domain gain of 18.24 points.
Deep Read · 2026-08-15高松灯 / Agent 日报
RLGRPOSelf-DistillationRouting