OpenAI revealed that Sora treats video as spacetime patches inside a Diffusion Transformer (DiT), catalyzing an industry-wide migration to DiT across open and closed image/video models—including FLUX by Black Forest Labs and spatiotemporal Video Generation Models (Sora, Veo, Kling, Wan) using 3D VAEs.
Diffusion Transformer (DiT)
Also known as: DiT · MMDiT
Plain English
An architecture that swaps out the older convolutional U-Net inside image and video generators for a Transformer—bringing the same massive scaling laws that powered LLMs into visual generation.
Technical Overview
Proposed by Peebles and Xie, DiT patchifies noisy latent tensors from a VAE into visual tokens and processes them with Transformer blocks conditioned via adaptive layer normalization (adaLN-Zero). Multimodal DiT (MMDiT) variants jointly attend over text and visual tokens, forming the backbone of FLUX and modern Video Generation Models (Sora, Veo, Kling, Wan).
Chronological Entries (2)
William Peebles and Saining Xie demonstrated that replacing the convolutional U-Net in Diffusion Models with a standard Transformer operating on latent patches yields predictable scaling gains as model GFLOPs increase.
Direct Relations (3)
- ImplementsTransformer
- Part ofDiffusion Models
- UsesLatent Space & VAE