From Pixels to States: World Models Should Think Like Game Engines

The number one paper on HuggingFace Daily Papers, at a huge 407 upvotes: From Pixels to States: Rethinking Interactive World Models as Game Engines (arXiv 2607.14076, from Alaya Studio). The argument is simple and overdue. Interactive world models today mostly predict pixels — the next frame, then the next. A game engine does not work that way: it maintains explicit state, applies actions to state, and renders observations from state. The paper says world models should do the same, and organizes the whole field along four dimensions: action control, state dynamics, state-observation persistence, and real-time generation.

The concrete contribution is a dataset: 90-plus hours of Black Myth: Wukong gameplay with frame-aligned player actions, ground-truth game states pulled from the engine, and visual observations. That triple alignment — action, true state, pixels — is exactly what pixel-only video datasets cannot give you, and it makes state-aware world modeling trainable and measurable for the first time at this scale.

Why this belongs here: it is the same debate agent builders are having about memory. Do you store the rendering, or the state that generates the rendering? Every team that moved from chat-log memory to structured state made the same bet this paper makes. No code released yet — watch the repo situation.

Paper: https://arxiv.org/abs/2607.14076