统一框架 = 集中训练分布式执行（CTDE）+ Hybrid Reward Architecture（HRA）。多 actor 共享集中式多头 critic，分解为 task 头（稀疏任务奖励）与 grasp 头（potential-based 抓取奖励）；重构 critic 与 actor 目标利用分解 Q 值，显式处理离散夹爪策略的分类动作分布。
章节结构（全文标题提取）： I INTRODUCTION II RELATED WORK III METHODOLOGY IV EXPERIMENTAL SETUP AND RESULTS V CONCLUSIONS · II-A Real-World Online Reinforcement Learning · II-B Hybrid Action Space with Multi-Agent RL · III-A Sample-Efficient RL with Prior Data · III-B Multi-agent RL with Hybrid Action Space and CTDE · III-C Decomposed Critic via HRA · III-D Compliant Gripper Design · IV-A Policy Training Configuration · IV-B Experimental Setups and Task Design Overview · IV-C Experiment Results
Real-world online reinforcement learning (RL) provides a promising approach for training robotic manipulation policies directly in the physical world, avoiding the sim-to-real gap and enabling continuous policy refinement through human-in-the-loop interaction. Recent methods have demonstrated sample-efficient learning through human intervention but remain limited to small randomization ranges and encounter challenges with the non-stationarity induced by concurrently training multiple agents. To address these limitations, we introduce a unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA). This enables multiple actors to share a centralized multi-head critic. The critic is decomposed into task and grasp heads, corresponding to the sparse task reward and a potential-based grasping reward, respectively. We accordingly reformulate the critic and actor objectives to exploit the decomposed Q-values while explicitly accounting for the categorical action distribution of the discrete gripper policy. Experimental results demonstrate that the proposed framework substantially improves both sample efficiency and policy performance. We validate our approach on two robotic arms and a simulated humanoid robot across tennis ball and banana pick-and-place, pot reset, and simulated block relocation tasks under dimension-wise domain randomization, approximately 5-25x larger than those considered in prior work. Compared with a state-of-the-art baseline, our method improves the success rate from 60% to 80% on tennis ball pick-and-place, from 60% to 90% on banana pick-and-place, and from 25% to 95% on simulated block relocation, while also successfully accomplishing a task where the baseline consistently fails. Videos and more details are available at our project website: this https URL .