FIFoundationIndex
IndexMapAbout
ArchitectureCross-ModalLLMImageVideo

Transformer

Also known as: Self-Attention · Decoder-Only Transformer

Plain English

The core neural network architecture behind modern LLMs and increasingly image and video generators. Instead of reading data strictly step-by-step like older recurrent networks, a Transformer compares every piece of an input sequence against every other piece in parallel.

Technical Overview

Introduced in 2017, the Transformer replaces recurrence with stacked multi-head self-attention and feed-forward blocks paired with residual connections and normalization. Modern frontier LLMs predominantly use causal decoder-only variants with RoPE, FlashAttention & GPU Kernels, and often Mixture of Experts (MoE) layers, while image and video models adopt the architecture via DiT.

Chronological Entries (3)

#1Paper
Curated

Vaswani et al. at Google published Attention Is All You Need, introducing scaled dot-product self-attention as a parallelizable replacement for RNNs and LSTMs. Because attention computes pairwise interactions across all tokens simultaneously, it unlocked massive GPU scaling during training.