← 首页|学术|How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent RL
cs.AI · 2608.07118 · 2026-08-07

How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

Lichao Ma,Yang Sun,Shuaitao Zhao,Yangyi Fang,Cong Qin,Xiaoliang Fu,Yuhang Tian,Yuchen Wei,Junbo Zhu,Yang Wei,Lu Pan,Jiaye Lin
TL;DR:多轮 agent RL 的信用分配在两层进行:轨迹级→动作,动作级→token。FACTOR 把这两层决策分开,用 TD 残差做动作信用、用 teacher-student 似然差距做 token 分配,显著提升 agent RL 效率。

🎯 问题

多轮 agent RL 的信用分配有两个层面:轨迹级信用要分给动作,每个动作的信用还要分到各 token。现有方法混在一起做,导致两层的信号互相污染。

🔬 方法

FACTOR 三组件:TAC(TD Action Credit,用 checkpoint 校准的 TD 残差给动作分配信用,telescope 到轨迹优势)、HTA(Hindsight Token Allocation,用反馈条件下的 teacher-student 似然差距分配 token 信用)、APM(Per-Action Mean Preservation 保持分配守恒)。
章节结构(全文标题提取):
1 Introduction
2 Preliminaries
3 FACTOR
3.1 TAC: TD Action Credit
3.2 HTA: Hindsight Token Allocation
3.3 APM: Per-Action Mean Preservation
3.4 Action-Mean Loss and Conservation
4 Experiments
4.1 Experimental Setup
4.2 Main Results

📊 结果

在 ALFWorld / WebShop / ScienceWorld 上超过现有 agent RL 基线,消融与机制证据说明两层分离分配是有效性的关键。

📝 原文摘要

▶ 原文摘要 Abstract
Credit assignment in multi-turn agent reinforcement learning operates at two levels: assigning trajectory-level credit to actions and distributing each action's credit across its tokens. In this paper, we introduce FACTOR, which separates these decisions. FACTOR uses checkpoint-calibrated TD residuals to assign per-action credits that telescope to the trajectory advantage, and feedback-conditioned teacher-student likelihood gaps to allocate each credit across the realized action tokens. Per-action normalization preserves the action-average coefficient and prevents token-level sign flips. We pair this construction with an action-mean reduction, removing the implicit dependence of an action's scalar surrogate weight on its token length. At the behavior policy and before clipping, each action's inner action-mean surrogate equals its TD credit. FACTOR consistently improves over competitive baselines across ALFWorld, WebShop, and ScienceWorld, with every environment-seed comparison favoring FACTOR and the largest gains emerging on the longest-horizon environment. The same hyperparameters transfer without retuning to a larger backbone and to a different model family. Ablations identify TD action credit as the dominant driver of the improvement, with hindsight token allocation contributing complementary gains.
Deep Read · 2026-08-11高松灯 / Agent 日报
AgentRLCredit-AssignmentMulti-Turn