← 首页|学术|A Framework for Designing Reward Functions: From Objectives to Features to Human-Aligned Reward Functions
cs.LG · 2608.12302 · 2026/08/12

A Framework for Designing Reward Functions: From Objectives to Features to Human-Aligned Reward Functions

Shi, Di Yang, Knox, W. Bradley
TL;DR:面向非专家的形式化奖励函数设计流程:把自然语言任务目标蒸馏成基础目标→推导可测结果变量→用因果DAG选代表性子集→偏好elicitation拟合权重,并保持确定性无冲突的可行权重域。

🎯 问题

设计符合人类偏好序的奖励函数对非专家很难,缺少可实例化、可迭代的正式流程。

🔬 方法

三步:1)引导工作流推导结果变量;2)奖励项选择降为因果DAG上最小成本部分覆盖(max-flow多项式求解);3)权重拟合几何化为凸可行域,由separation oracle在O(n log κ)偏好查询内迭代收窄。
章节结构(全文标题提取):
1 Introduction
2 Related Work
3 Method
4 Conclusion
References
Appendix A Reduction of Minimum Cost Partial Cover
3.1 Selecting Outcome Variables from Task Objectives
3.2 A Causally-Aware Outcome Variable Selection Framework
3.3 Fitting the Reward Function’s Parameters to Preferences
A.1 Preliminaries: Flow Networks (8)
A.2 Structural Characterization
A.3 Reduction to Min-Cut Proof

📊 结果

首个保持确定性无冲突可行权重域、可通过separation oracle收敛到期望容差的奖励设计方法。

💡 与研究方向关联

奖励设计的形式化流程:目标→结果变量→因果子集→偏好拟合,保持无冲突可行权重域。属用户 Reward Design 兴趣方向。

📝 原文摘要

▶ 原文摘要 Abstract
We present a formal process to enable non-experts to instantiate and iterate on human-aligned reward functions, i.e. reward functions that adhere to a given preference ordering over trajectories. Given a task described in natural language, our process produces a linear reward function in three steps: distill the task's objectives into a set of fundamental objectives and derive measurable outcome variables that capture those fundamental objectives, select a causally representative subset of outcome variables as the reward terms, and fit weights to those reward terms via preference elicitation. Our contributions describe the first step and formalize the latter two steps. The first is a guided workflow for deriving outcome variables. The second is a reduction of reward term selection to minimum-cost partial cover on a causal DAG, solved in polynomial time via max-flow. The third is a geometric framing of weight fitting as a convex feasibility problem iteratively narrowed by preference queries, solved by existing separation oracle methods. To the best of our knowledge, this is the first reward-design method that maintains a deterministically conflict-free feasible weight region, narrowed to a desired tolerance via a separation oracle with O(n log \kappa) preference queries.
Deep Read · 2026-08-14高松灯 / Agent 日报
Reward DesignReward FunctionPreference ElicitationCausal DAG