By 2025, the Transformer backbone unified text, image, and video generation: visual generators replaced U-Nets with DiT backbones (such as FLUX and modern Video Generation Models (Sora, Veo, Kling, Wan)), allowing the same GPU scaling laws and FlashAttention & GPU Kernels kernels to serve every modality.
Transformer
Also known as: Self-Attention · Decoder-Only Transformer
Plain English
The core neural network architecture behind modern LLMs and increasingly image and video generators. Instead of reading data strictly step-by-step like older recurrent networks, a Transformer compares every piece of an input sequence against every other piece in parallel.
Technical Overview
Introduced in 2017, the Transformer replaces recurrence with stacked multi-head self-attention and feed-forward blocks paired with residual connections and normalization. Modern frontier LLMs predominantly use causal decoder-only variants with RoPE, FlashAttention & GPU Kernels, and often Mixture of Experts (MoE) layers, while image and video models adopt the architecture via DiT.
Chronological Entries (3)
Decoder-only Transformers converged on a standard modern recipe—RMSNorm, SwiGLU feed-forward blocks, Grouped-Query Attention (GQA), and RoPE—popularized by open-weight releases like Llama and served efficiently with KV Cache and FlashAttention & GPU Kernels.
Vaswani et al. at Google published Attention Is All You Need, introducing scaled dot-product self-attention as a parallelizable replacement for RNNs and LSTMs. Because attention computes pairwise interactions across all tokens simultaneously, it unlocked massive GPU scaling during training.
Direct Relations (4)
- UsesTokenization
- UsesFlashAttention & GPU Kernels
- EnablesKV Cache
- EnablesDiffusion Transformer (DiT)