Ha and Schmidhuber published World Models, pairing a VAE vision encoder with a recurrent dynamics predictor so reinforcement-learning agents could train policies entirely inside a learned latent dream environment.
World Models
Also known as: Interactive World Models · Genie · Dreamer
Plain English
AI models that learn an internal simulator of how an environment works—predicting future visual frames and physics in response to user actions so agents or players can interact in real time.
Technical Overview
World models learn compressed latent dynamics conditioned on actions, enabling policy training in imagination (Dreamer) or real-time interactive video simulation (Google DeepMind's Genie series). Modern generative world models combine spatiotemporal Transformers, diffusion, and autoregressive frame prediction with synchronized audio and low-latency streaming.
Timeline (4)
2018
- Paper
2023
- Paper
Hafner et al. introduced DreamerV3, a general-purpose model-based RL agent using a discrete-latent world model that mastered diverse domains—including collecting diamonds in Minecraft from scratch—with fixed hyperparameters.
2024
- Paper
Bruce et al. at Google DeepMind introduced Genie, an 11B-parameter generative interactive environment trained from unlabeled internet videos that turns single image prompts into frame-by-frame playable 2D worlds.
2026
- Release
Google DeepMind rolled out Genie 4 inside Project Genie, adding native synchronized audio and live natural-language scene editing without restarting the real-time interactive simulation.