← 首页|学术|The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior
cs.SE · 2608.11386 · 2026/08/11

The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior

Xu, Xiangzhe, Saghir, Hamidreza, Wu, Qianhui, Côté, Marc-Alexandre, Wang, Tong, Lakkaraju, Kiran, Pei, Kexin, Zhang, Xiangyu
TL;DR:工具架构(能力如何组织与暴露给模型)显著改变编码Agent行为:同样的能力,结构化低层接口使重复尝试一致性提升最高4.7×,Python CodeAct式接口以41.6%更少步骤、56.3%更低token达到相似性能。

🎯 问题

此前工具研究聚焦「扩展agent能做什么」,却少系统关注能力如何组织与暴露(tool architecture)这一设计维度。

🔬 方法

仓库级issue修复受控实验:6种工具架构在信息与动作相似、仅组织与暴露方式不同,跨3个actor共11,700条轨迹。
章节结构(全文标题提取):
1 Introduction
2 Setups
3 Evaluation Results
4 Related Work
5 Conclusion
Acknowledgments
LLM Usage Disclosure
References
2.1 Tool Setups
2.2 Evaluation Protocol
2.3 Evaluation Dimensions and Metrics
3.1 Task Resolve Rate
3.2 Consistency
3.3 Exploration
3.4 Efficiency
3.5 Why Cognitive Scaffolding Has Limited Effects
3.6 Generalization to Additional Coding Tasks
B.1 Performance
B.2 Consistency Results with Additional Repeats
B.3 Mechanism Analysis: Atomic Reduces Environment-Interaction Errors

📊 结果

结构化低层接口提升一致性最高4.7×;自然语言搜索拓宽探索、相关文件访问提升11%+;CodeAct式接口相似性能但-41.6%步骤/-56.3% token;轻量文本认知脚手架工具(记录中间推理)对行为影响有限。

💡 与研究方向关联

工具架构(能力如何组织暴露)独立于能力集合影响 agent 行为——这是 agent 系统设计的核心杠杆,与用户关注的 agent 架构直接相关。

📝 原文摘要

▶ 原文摘要 Abstract
As large language models continue to improve, agentic systems are becoming increasingly important, and tools are a key design dimension because they determine how agents access information and take action in their environments. Prior work on agent tooling has primarily focused on expanding what agents can do, but has paid less systematic attention to how those capabilities are organized and exposed to the model. We refer to this latter design dimension as tool architecture. We study tool architecture in coding agents through controlled experiments on repository-level issue fixing, comparing six tool architectures that hold the underlying information and actions similar while varying how they are organized and exposed to the model, across three actors and a total of 11,700 trajectories. Our experiments show that, even when tools provide similar capabilities, tool architecture changes agent behavior: Compared to a basic architecture where the agent has only the bash tool, more structured low-level interfaces improve consistency across repeated attempts by up to 4.7 $\times$; natural-language search broadens repository exploration and increases access to relevant files by more than 11%; and Python CodeAct-style interfaces achieve similar task performance with 41.6% fewer steps and 56.3% lower token usage. By contrast, lightweight text-based cognitive-scaffolding tools, such as tools that let the agent record intermediate reasoning, have limited effect on actor behavior.
Deep Read · 2026-08-14高松灯 / Agent 日报
Tool ArchitectureCoding AgentCodeActAgent Design