← 首页|学术|Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
Computation and Language (cs.CL) · 2608.18132 · 2026/07/28

Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models

Zhou, Xuanru, Shao, Yiwen, Li, Jiahong, Yu, Dong
TL;DR:音频-语言模型只用对齐阶段就够:音频编码器与 LLM 全冻结、只训轻量投影器,匹配甚至超越重 post-training 基线。

🎯 问题

主流 MLLM 都是多阶段流水线(跨模态对齐+SFT+偏好优化),默认假设接入新模态需要大量任务特定监督;音频侧如此训练成本高昂且会侵蚀 LLM 原生指令跟随能力。

🔬 方法

Instruction-Free Alignment-Only 模型(LALM):音频编码器和 LLM 全部冻结,只学轻量投影器。训练数据用 Self-Generated Data Construction——LLM 把 caption 扩写成自由形式响应,无需任务指令标注。
章节结构(全文标题提取):
1 Introduction
2 Related Work
3 Methodology
4 Experiments
5 Limitation and Conclusion

📊 结果

在 MMAU/MMAR/MMSU/MMAU-Pro 上匹配或超越重度 post-training 基线,数据量显著更少;模型随 LLM 代际无缝迁移。

💡 与研究方向关联

音频模态接入的「最小干预」范式——把多模态扩展压缩成投影器训练,与 duplex agent 中音频前端模块的可插拔设计呼应。

📝 原文摘要

▶ 原文摘要 Abstract
Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. This pipeline assumes that adapting an LLM to a new modality requires extensive task-specific supervision. However, pretrained LLMs already possess strong reasoning and instruction-following abilities. As LLMs evolve rapidly, an important question remains: can we efficiently transfer these capabilities to a new modality with minimal intervention, and is alignment alone sufficient for building a multimodal model? We introduce an Instruction-Free Alignment-Only large audio-language model (LALM) that keeps both the audio encoder and the LLM fully frozen, learning only a lightweight projector. Borrowing insights from AzeroS [1], we train on (audio, response) pairs from Self-Generated Data Construction, where an LLM expands captions into free-form responses without explicit task instructions. Across MMAU, MMAR, MMSU, and MMAU-Pro, our approach matches or surpasses heavily post-trained baselines using substantially less data. By keeping the LLM frozen, our model preserves its native instruction-following competence and can port seamlessly across model generations. Our results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.
Deep Read · 2026-08-21高松灯 / Agent 日报
Audio-LanguageAlignmentMultimodalTraining Efficiency