IndexMapAbout
ArchitectureLLMImageVideoCross-Modal

Attention

Also known as: Self-Attention · Multi-Head Attention · MHA · GQA · Grouped-Query Attention · Multi-Head Latent Attention · MLA

Plain English

The mechanism that lets a neural network weigh which parts of an input sequence matter most when processing each token, replacing slow step-by-step recurrence with parallelizable all-to-all context.

Technical Overview

Scaled dot-product attention computes relevance weights between query, key, and value projections via softmax(QK^T / sqrt(d_k))V. Multi-Head Attention (MHA) runs parallel projection subspaces, while modern inference optimizations—Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and FlashAttention—reduce KV cache memory bandwidth and HBM I/O bottlenecks.

Timeline (5)

RSS

2014

2017

  1. Paper

    Vaswani et al. published Attention Is All You Need, replacing recurrent layers entirely with Multi-Head Self-Attention (softmax(QK^T / sqrt(d_k))V) inside the Transformer, enabling massive GPU parallelism during pre-training.

2022

2023

2024

See also (bkz.) (12)

Open full graph
7 more connections