← 首页|学术|Social Gym: Verifiable Multi-Agent Social Skill Benchmark
cs.CL cs.AI cs.MA · 2608.09128 · 2026-08-10

Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

Keyu He, Xuhui Zhou, Maarten Sap
TL;DR:LLM 社交推理难以评测(无客观 ground truth,LLM judge 昂贵主观噪声)。Social Gym:21 个多 agent 社交游戏(狼人杀/抵抗组织/间谍等),规则决定的结果可验证,Elo 锦标赛跨游戏排行榜。SPaRTan 训练-free 自改进循环(玩-反思-迁移 playbook)。

🎯 问题

LLM agent 越来越多部署在需要合作、协商、适应其他 agent 的多 agent 社交环境。社交技能评测难:不像数学逻辑有客观 ground truth,评测退回 LLM judge(昂贵、主观、噪声),模型也无可靠信号学习。

🔬 方法

Social Gym 环境:21 个多 agent 社交游戏(狼人杀、抵抗组织、间谍等),规则决定的结果让表现可验证客观,Elo 锦标赛产生跨游戏排行榜。SPaRTan(Self-Play and Reflect-Transfer):训练-free 自改进循环——模型玩游戏、反思轨迹与结果产出可迁移 playbook、在后续游戏应用。
章节结构(全文标题提取):
1 Introduction
2 Related Work
3 Social Gym Benchmark
4 SPaRTan: Self-Play and Reflect-Transfer
5 SPaRTan Experiments and Results
6 Conclusion and Discussion
Limitations
Ethics / Broader Impacts
Acknowledgments
· 3.1 System Architecture
· 3.2 Game Suite
· 3.3 Measuring success via Elo Tournament
· 3.4 Leaderboard Results
· 5.1 Same-model reflection
· 5.2 Cross-model distillation
· 5.3 Open-weights model replication
· G.1 GPT-5-mini Werewolves Playbooks
· G.2 GPT-5 Werewolves Playbooks
· I.1 Cross-play role gaps

📊 结果

GPT-5-mini 登顶排行榜,但无模型所有游戏/所有角色全能;SPaRTan playbook 帮 GPT-5-mini 提升弱角色表现,但大体不提升 Qwen3-32B。

📝 原文摘要

▶ 原文摘要 Abstract
LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and improving these social skills is hard because, unlike math or logic, social interaction offers no objective ground truth: evaluations fall back on LLM judges, which are costly, subjective, and noisy, and models get no reliable signal to learn from. To address both, we first introduce Social Gym, an environment of 21 multi-agent social games (e.g., Werewolves, Resistance, Spyfall) whose rule-decided outcomes make agent performance verifiable and objective, with an Elo tournament that produces a cross-game leaderboard. Benchmarking experiments show that while GPT-5-mini tops the leaderboard, no model excels at all games uniformly or in all game roles, pointing to limitations of social reasoning. Motivated by this, we additionally propose SPaRTan (Self-Play and Reflect-Transfer), a training-free self-improvement loop: a model plays a game, reflects on its trajectories and their outcomes to produce a transferable playbook, and applies that playbook in subsequent games. Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance. Together, Social Gym and SPaRTan offer a reproducible, verifiable foundation for measuring and improving LLM social reasoning without weight updates.
Deep Read · 2026-08-12高松灯 / Agent 日报
Multi-AgentSocialBenchmarkSelf-Play