← 首页|学术|检测到效应≠学会行动:LLM Acquisition Agent 的 Reward-SNR 下限
cs.LG / cs.CL / cs.IR · 2608.10441 · 2026/08/11

Detecting an Effect Is Not Learning to Act on It: A Reward-SNR Floor for LLM Acquisition Agents

Yuan, Ying
TL;DR:「检测到平均有效」≠「能逐实例学会使用」:当奖励 SNR ρ 不高于 ρ*(N)≈2.8/√N,任何可部署策略都学不会何时获取信号——连序统计噪声 placebo 都能复现 ≥100% 的表观增益。给 LLM acquisition agents 一个可检测性下限。

🎯 问题

许多流水线可付费获取辅助观测(LLM 结构化推理、慢 oracle、昂贵测量),必须决定何时值得使用——但 in-sample oracle 的表观增益可能只是噪声的序统计。

🔬 方法

核心区分「检测平均效应」vs「学习逐实例获取策略」;导出 reward-SNR detectability floor ρ*(N)≈2.8/√N;正对照(positive control)确认这是真实的低 SNR 极限而非流水线缺陷。
章节结构(全文标题提取):
1 Introduction
2 Problem Setup
3 Method: Structured Hypothesis Embeddings (SHE)
4 Agent-Side Quality of the Hypotheses
5 Downstream Value Is Backbone- and Regime-Conditional
6 Learning When to Acquire Fails at Every Granularity
7 The Mechanism: a Three-Dataset Reward-SNR Story
8 The Reward–SNR Detectability Floor
9 Related Work
10 Discussion, Deployment, and Limitations
5.1 A redundancy gradient over baseline strength · 5.2 A controlled ordered-vs-unordered test

📊 结果

跨 per-impression/cluster/regime/uplift-tree 各粒度,learned routing 从不优于随机;匹配矩噪声 placebo 复现 ≥100% 的 oracle 表观增益;引入 SHE(Structured Hypothesis Embeddings,冻结 LLM 把用户历史转成排序/置信度/证据支撑的嵌入)。

💡 与研究方向关联

给 RL/决策流水线一个反直觉的警告:平均效应的可检测性不等于逐实例可学习性。SNR 下限 2.8/√N 是判断「这条路能不能学会」的先验判据,避免在噪声上浪费训练。

📝 原文摘要

▶ 原文摘要 Abstract
Many pipelines can pay a per-example cost to acquire an auxiliary, model-derived observation -- an LLM's structured reasoning, a slow oracle, an expensive measurement -- and then must decide when the acquired signal is worth using. Our thesis is a distinction that is easy to miss: detecting that such a signal helps on average is not the same as learning to act on it per instance, and a reward-SNR floor governs when the second is even possible. Even when the signal is faithful and an in-sample oracle picking the top-b examples by realized reward shows a sizable apparent gain, no deployable policy can learn when to acquire it: across per-impression, cluster, regime, and uplift-tree granularities, learned routing never beats random, and a matched-moment noise placebo reproduces >=100% of the oracle's apparent gain -- the apparent "learnable structure" is order statistics of noise. We explain this with one distinction, detecting a mean effect vs. learning a per-instance acquisition policy, and a reward-SNR detectability floor: routing is estimable offline only if the reward SNR rho clears rho*(N) ~= 2.8/sqrt(N), with a positive control confirming a true low-SNR limit rather than a broken pipeline. As a concrete instantiation we introduce Structured Hypothesis Embeddings (SHE): a frozen LLM turns a user history into ranked, confidence-scored, evidence-grounded intent hypotheses, fused into a recommender. On three public datasets (MIND, REES46, Amazon-Beauty), SHE is faithful and calibratable, yet its value is backbone- and regime-conditional (significant over an ordered GRU, +0.0114, 95% CI [+0.0030, +0.0209], but a global redundancy gap indistinguishable from zero), and learned acquisition collapses at every granularity because all three datasets sit below the floor. The realizable unit is a design-time regime gate, not a per-instance policy. We release code and a one-command reproduction.
Deep Read · 2026-08-13高松灯 / Agent 日报
RLRewardSignalTheory