← 首页|学术|POLIS: Institutional Safety in Multi-Agent Systems
cs.LG cs.AI cs.MA · 2608.09828 · 2026-08-10

Multi-Agent AI Safety as an Institutional Design Problem

Abdullah X
TL;DR:AI agent 运行在治理其委托/信息/动作/共享资源的制度内。POLIS 首篇论文:5,280 集研究套件揭示同一最终违规率可隐藏不同机制——宪法提示与 provenance-aware guard 各 0/384 违规,local-state guard 在 laundering 场景 22/96 违规(provenance 0/96)。制度本身是安全的一部分。

🎯 问题

AI agent 越来越多运行在治理其委托任务、移动信息、执行动作、使用共享资源的系统内。问题:AI 制度的哪些部分产生安全、如何产生?同一违规率下机制可以完全不同。

🔬 方法

POLIS 研究项目首篇。冻结 5,280 集研究套件:预定委托实验跨 4 个模型家族 + 高冲突诊断加 3 个端点;对比不同规则表述、guard 咨询不同权威状态、变化合规内部 fallback 吸引力、允许阻塞工作流继续。
章节结构(全文标题提取):
1 Introduction
2 Experimental Design
3 Delegation Results
4 Resource Commons: Institutions Also Act Through Information
5 Related Work
6 Discussion and Limitations
7 Conclusion
· 2.1 Core delegation environment
· 2.2 Institutional treatments and outcome definitions
· 2.3 Study matrix and analysis
· 3.1 Institutions change behavior when compliance becomes costly
· 3.2 Local-guard failures concentrate in mutable-state laundering scenarios
· 3.3 Prompt compliance and executable containment are different safety mechanisms

📊 结果

详细宪法提示 0/384 实现违规;provenance-aware guard 也 0/384(51/384 阻塞禁止尝试,44/51 后来安全完成)。local-state guard 失败集中在普通变换改变可见策略但权威源不变处,laundering 场景 22/96 违规 vs provenance 0/96(p=4.77e-7)。资源分配实验显示揭示数值改变 agent 请求。

📝 原文摘要

▶ 原文摘要 Abstract
AI agents increasingly work inside systems that govern how they delegate tasks, move information, execute actions, and use shared resources. Recent work already shows that deployment rules can change collective behavior. Here we ask which parts of an AI institution produce safety and how they do it. This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems. We report a frozen 5,280-episode study suite. The main pre-specified delegation experiment spans four model families; a targeted high-conflict diagnostic adds three additional model endpoints. In matched structured workflows, the model sees different rule formulations and guards consult different authority states. We also vary the attractiveness of the immediate compliant internal/self fallback and allow blocked workflows to continue. A detailed constitutional prompt produces 0/384 realized violations. A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely. The local-state guard's failures concentrate in scenarios where an ordinary transformation changes visible policy while originating authority stays fixed. In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7). A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests. In these structured workflows, the same final violation rate can hide very different mechanisms. The rule itself is only part of the institution. The authority state the system trusts matters, and so does the path available after a block.
Deep Read · 2026-08-12高松灯 / Agent 日报
Multi-AgentSafetyInstitutionGovernance