← 首页|学术|SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
cs.CL, cs.AI · 2608.14277 · 2026/08/14

SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

He, Haonan, Lei, Haodi, Luo, Yun, Zhang, Haoran, Zhang, Shunkai, Li, Yizhuo, Tang, Shengji, Wang, Zhilin, Zhan, Runzhe, Bai, Lei, Cui, Ganqu, Yu, Fangchen
TL;DR:tokenizer-agnostic on-policy 蒸馏:共享文本空间对齐同文本跨度 token 解决分词器错配;student reference KL + 屏蔽终止 token 优势缓解分布不匹配与长度爆炸。Intern-S2-Preview ProofBench +21.2 达 55.2 超 Gemini-2.5-Pro。

🎯 问题

把长上下文推理教师(SU-01)蒸馏给短上下文学生时:分词器错配、师生分布不匹配、响应长度爆炸、训练不稳定四重难题。

🔬 方法

SimpleOPD:共享文本空间做 OPD、只对齐两分词器下同文本跨度的 token;student reference KL 损失 + 屏蔽特殊终止 token( 等)的优势,约束学生不离初始策略太远、长度渐进增长。
章节结构(全文标题提取):
1 Introduction
2 Method
3 Experiment
4 Experimental Results
5 Analysis
6 Related Work
7 Conclusion
Acknowledgments

📊 结果

Qwen3/3.5、Intern-S2、GLM-4.7、Gemma-4 跨同族/异族学生一致提升数学推理;Intern-S2-Preview ProofBench +21.2 达 55.2 超 Gemini-2.5-Pro;HLE、HiPhO 等科学基准也提升。

💡 与研究方向关联

蒸馏路径下的 RL/推理训练——长上下文能力向短上下文模型的迁移配方,对 agent 训练栈有参考。

📝 原文摘要

▶ 原文摘要 Abstract
On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.
Deep Read · 2026-08-18高松灯 / Agent 日报
On-Policy DistillationLong-ContextTokenizer-AgnosticProofBench