← 首页|学术|Ex-Omni-2D: 表现力全模态对话与原生视觉呈现
cs.AI / cs.CL / cs.CV · 2608.10720 · 2026/08/11

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Zhang, Haoyu, Li, Zhipeng, Tang, Xiaoying, Yu, Tianshu, Guo, Yiwen
TL;DR:全模态对话模型生成「文本 + 个性化语音 + 参考条件视频」的协调响应:Visual Thought Plan 描述场景/情绪/动作,语音单元作为共享声学-时间接口与视频帧在线对齐,蒸馏出 Streaming Student 实现 RTF=1.293 的低延迟增量生成。

🎯 问题

全模态对话模型能理解多模态输入并合成语音回复,但响应是「视觉上无实体」的——缺少与语音协调的面部/身体呈现。

🔬 方法

预测结构化 Visual Thought Plan(场景/情绪/动作)→ 响应文本 + 原生多码本语音单元;语音单元构成共享声学-时间接口,解码为语音并与视频帧在线对齐,让响应路径与 avatar 路径可从异构数据分开学习;全序列 Video Generator 作 Teacher,蒸馏出四步 block-causal Streaming Student,Prefix Streaming 跨 chunk 携带干净隐状态减少累积退化。

📊 结果

四步推理下四 GPU 端到端 RTF=1.293 @400×720,给出实用的质量-效率操作点。

💡 与研究方向关联

表现力数字人 / 全双工语音助手需要「语音 + 表情 + 动作」同步。Ex-Omni-2D 的共享声学-时间接口避免了对大规模 query-text-speech-video 四元组监督的依赖,把响应与呈现两条路径解耦训练。

📝 原文摘要

▶ 原文摘要 Abstract
Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce \textbf{Ex-Omni-2D}, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured \textit{Visual Thought Plan} (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal \emph{Streaming Student} whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at $400\times720$/$720\times400$, providing a practical quality--efficiency operating point.
Deep Read · 2026-08-13高松灯 / Agent 日报
Omni-ModalDialogueAvatarSpeechVideo