← 首页|学术|Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
cs.LG,cs.CL · 2608.07371 · 2026-08-07

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

Haoyu Zheng,Yun Zhu,Qing Wang,Wenqiao Zhang
TL;DR:agentic RL 的 hindsight 信号太多、在轮次间怎么分配没标准。TRIAL 用"轨迹相对"的 hindsight 蒸馏 + 统一的 turn-aligned 打分协议,把每个决策轮的结果视图与 hindsight 条件视图对齐分配。

🎯 问题

agentic RL 常用 hindsight 补充稀疏结果奖励,但一次完整 rollout 会产生很多 hindsight 信号,跨轮次如何分配缺少统一协议。

🔬 方法

TRIAL(Trajectory-Relative Hindsight Distillation):对每个决策轮提取该决策"实际结果"的 outcome view,在同一响应下比较普通条件与 hindsight 条件的评估,得出 turn-aligned 的相对信用,再做联合优化。
章节结构(全文标题提取):
1 Introduction
2 Related Work
3 Method
3.1 Preliminaries
3.2 Trajectory-Relative Hindsight Allocation
3.3 Joint Optimization
4 Experiments
4.1 Experimental Setup
4.2 Overall Agentic Performance
4.3 Controlled Revision Profiles

📊 结果

在 agentic 基准上整体性能提升,受控修订画像显示 hindsight 分配更精准;附录给出完整训练协议与条件示例。

📝 原文摘要

▶ 原文摘要 Abstract
Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.
Deep Read · 2026-08-11高松灯 / Agent 日报
AgentRLHindsightDistillation