← 首页|学术|Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
cs.CV · 2608.24680 · 2026-08-25

Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training

Wenxuan Shen, Dongna Jin, Dongping Chen
World ModelsVideo GamesData Curation
💬 游戏画面里的 UI 界面会污染世界模型训练数据;G2WEngine 自动提取并合成/移除 UI 叠加层,用无 UI 画面训练的世界模型 VideoReward 提升 6.83%。

🎯 背景

电子游戏为视频世界模型提供了可扩展的训练数据来源——多样化的环境、复杂的交互、以及海量的野生 gameplay 视频。但原始录屏画面把游戏世界本身和屏幕空间的 UI 界面纠缠在一起,引入了游戏特定的偏差和无关动态,妨碍了世界模型的训练。

🔬 方法

论文提出 GameUI-Taxonomy 和 G2WEngine,一个形式化 gameplay UI 定位与移除任务的全栈框架。G2WEngine 能自动从真实 gameplay 视频中提取可复用的 UI 资产,并在干净画面上合成时间一致的 UI 叠加层。基于此构建了 Game2World 数据集,包含 9.6 万条带精确重建目标的合成配对视频,以及来自 303 款游戏的 1,079 条野生片段用于真实场景评测;资产库涵盖 21 个分类下的 5,132 个已验证 UI 元素。论文进一步提出 GameCleaner,一个无需 mask 的 gameplay UI 移除模型,结合多模态语义理解与视频编辑能力,直接识别并移除多样化的 HUD 元素,同时保留底层场景内容和时间动态。

📊 结果

在受控试点实验中,用无 UI 画面训练的世界模型相比用 UI 叠加画面训练的模型,VideoReward 提升 6.83%。在 UI 移除评测上,GameCleaner 在合成视频上取得平均 AAR 95.36,超过最强的时序 mask 基线 57.3%;在野生数据上取得最佳 AAR 80.05,背景保留率达到 99.8%。这些结果展示了把互联网 gameplay 视频转化为高质量世界模型训练数据的可扩展潜力。

原文摘要

▶ Abstract
Video games provide a scalable source of training data for video world models, offering diverse environments, complex interactions, and abundant in-the-wild gameplay videos. However, raw gameplay footage entangles the game world with screen-space interfaces, introducing game-specific biases and irrelevant dynamics that hinder world-model training. To address this problem, we introduce GameUI-Taxonomy and G2WEngine, a full-stack framework that formalizes gameplay UI grounding and removal. G2WEngine automatically extracts reusable UI assets from real gameplay videos and synthesizes temporally coherent UI overlays on clean footage. Using this engine, we construct Game2World, comprising 96K synthetic paired videos with precise reconstruction targets and 1,079 in-the-wild clips from 303 games for realistic evaluation. Its asset library contains 5,132 verified UI elements across 21 taxonomy categories, collected from 1,010 representative gameplay frames. Based on Game2World, we propose GameCleaner, a mask-free gameplay UI removal model that combines multimodal semantic understanding with video editing capabilities. Unlike mask-based methods, GameCleaner directly identifies and removes diverse HUD elements while preserving the underlying scene content and temporal dynamics. In a controlled pilot, world models trained on UI-free gameplay improve overall VideoReward by 6.83% over those trained on UI-overlaid data. On UI-removal evaluation, GameCleaner achieves an average AAR of 95.36 on synthetic videos, outperforming the strongest temporal mask baseline by 57.3%, and obtains the best in-the-wild AAR of 80.05 with 99.8 background preservation. These results demonstrate the scalable potential of transforming Internet gameplay videos into high-quality world-model training data. Code, dataset, and model will be available at this https URL .
来源: arXiv:2608.24680 · 精读由高松灯生成,基于摘要与 arXiv 页面信息