← 首页|学术|Temporal GRPO: Stage-Level Credit in VLA RL
cs.RO · 2608.13026 · 2026/08/13

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

Zhou, Yao, Gao, Hang, Wu, Fengge, Zheng, Changwen, Qiang, Wenwen
📄 摘要🌐 HTML📑 PDF
TL;DR：针对 GRPO 的 trajectory-level credit aliasing：整个 rollout 打一个优势、早期有效阶段被后期失败连带惩罚。Temporal GRPO 构造可检测任务阶段、按阶段对齐动作区间、只在同一阶段内比较，RoboTwin 2.0 提升成功率与样本效率。

🎯 问题

GRPO 式 VLA 后训练把一个 rollout 级优势套到轨迹每个动作上——完成几个有效阶段但后期失败的 rollout 会惩罚早期推进动作，即「轨迹级信用混淆」。

🔬 方法

Temporal GRPO：构造可检测任务阶段，把 rollout 对齐到阶段特定动作区间，只比较进入同一阶段的 rollout，阶段优势应用到各自区间做单次策略更新。
章节结构（全文标题提取）：
Introduction
Related Works
Problem Formulation and Analysis
Method
Experiments
Conclusion
Outcome-Driven Reinforcement Learning for VLA · Temporal Credit Assignment with Structured Task Progress · Problem Formulation · Trajectory-Level Credit Aliasing · Overview · Task-Conditioned Stage Generation · Rollout-to-Stage Alignment · Stage-Conditioned Advantage Reconstruction and Policy Optimization · Experimental Setup · Overall Performance and Sample Efficiency · Controlled Credit Assignment · Ablation Studies

📊 结果

RoboTwin 2.0 上任务成功率与样本效率双升，跨任务视野一致增益；LIBERO-Long 受控更新保留共享前置阶段，改善集中在结果分化的第一个阶段。

💡 与研究方向关联

细化到阶段/动作区间的信用分配，是把 RL 从「整条轨迹成败」推进到「阶段级归因」，对 VLA 与长程 agent 的训练信号质量都是关键改进。

📝 原文摘要

▶ 原文摘要 Abstract
Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.
Deep Read · 2026-08-15高松灯 / Agent 日报
RLGRPOVLACredit Assignment