← 首页|学术|TideRL: 就绪感知调度提升 Agentic RL Goodput
cs.LG / cs.DC · 2608.10402 · 2026/08/11

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

Ren, Yanyu, Wang, Xizheng, Liu, Xiao, Lv, Bowen, Zhang, Hanchen, Zhang, Shudan, Lai, Hanyu, Wang, Shuai, Chen, Li, Li, Dan
TL;DR:多轮 agentic RL 的 rollout 要反复暂停等外部环境、恢复时上下文增长、完成时间高度可变。TideRL 用就绪感知调度:Continuous Task Batching 保留 rollout 状态 + Resource-Aware Ref-Actor Pipelining + Elastic Resource Scaling。文本/多模态 agentic 工作负载上 RL 训练 goodput 最高提升 5.6×。

🎯 问题

agentic RL 中训练 goodput(训练吞吐)比 GPU 占用率更重要:GPU 等待和重复 prefill 重算是纯开销。

🔬 方法

CTB(Continuous Task Batching)保留有用 rollout 状态;RA²P(Resource-Aware Ref-Actor Pipelining)在解耦流式与同居聚合之间按就绪积压与到达间隔选择;ERS(Elastic Resource Scaling)用同一就绪信号在 rollout 与训练之间移动 rank。
章节结构(全文标题提取):
1 Introduction
2 Background and Motivation
3 TideRL Overview
4 Continuous Task Batching
5 Resource-Aware Ref-Actor Collaboration
6 Evaluation
7 Discussion
8 Related Work
9 Conclusion
2.1 Agentic Reinforcement Learning · 2.2 Key Challenges in Asynchronous Agentic RL · 3.1 Design Rationale · 3.2 System Architecture and Workflow · 4.1 Token-Aware Admission Control · 4.2 Semantic-Aware Pausing and Resuming · 5.1 𝐑𝐀 2 ​ 𝐏 \mathbf{RA}^{2}\mathbf{P} Execution Strategies · 5.2 ERS Scheduling of 𝐑𝐀 2 ​ 𝐏 \mathbf{RA}^{2}\mathbf{P} Strategies

📊 结果

文本与多模态 agentic 工作负载:goodput 最高 +5.6×(vs 同步基线)、+33%(vs 异步基线),任务性能相当;KV cache hit 率 +1.58×,每步训练时间 -44.3%,总等待时间 -77.6%。

💡 与研究方向关联

agentic RL 与经典 RL 的基建差异:rollout 因外部工具/环境调用而高度异步。就绪感知调度(readiness-aware)把「何时算完」的差异显式纳入调度,是 agentic 训练基础设施的关键优化。

📝 原文摘要

▶ 原文摘要 Abstract
Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, $\textrm{RA}^2\textrm{P}$ selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6$\times$ over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58$\times$, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.
Deep Read · 2026-08-13高松灯 / Agent 日报
RLAgenticSchedulingInfra