Bahdanau et al. published Neural Machine Translation by Jointly Learning to Align and Translate, introducing additive attention over encoder hidden states so sequence-to-sequence decoders could dynamically focus on relevant source words instead of compressing an entire sentence into a fixed-length vector.
Attention
Also known as: Self-Attention · Multi-Head Attention · MHA · GQA · Grouped-Query Attention · Multi-Head Latent Attention · MLA
Plain English
The mechanism that lets a neural network weigh which parts of an input sequence matter most when processing each token, replacing slow step-by-step recurrence with parallelizable all-to-all context.
Technical Overview
Scaled dot-product attention computes relevance weights between query, key, and value projections via softmax(QK^T / sqrt(d_k))V. Multi-Head Attention (MHA) runs parallel projection subspaces, while modern inference optimizations—Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and FlashAttention—reduce KV cache memory bandwidth and HBM I/O bottlenecks.
Timeline (5)
2014
- Paper
2017
- Paper
Vaswani et al. published Attention Is All You Need, replacing recurrent layers entirely with Multi-Head Self-Attention (
softmax(QK^T / sqrt(d_k))V) inside the Transformer, enabling massive GPU parallelism during pre-training.
2022
- Paper
Dao et al. introduced FlashAttention, tiling attention matrix multiplications inside GPU SRAM so the quadratic
N × Nattention matrix never materializes in HBM—unlocking longer context windows with exact numerical equivalence.
2023
2024
See also (bkz.) (12)
Open full graphAttention