Gu, Goel, and Ré published S4 (Structured State Spaces for Sequence Modeling), showing that HiPPO-initialized linear state-space layers could model 16K+ long-range sequences without quadratic attention complexity.
State Space Models & Mamba
Also known as: Mamba · Mamba-2 · State Space Models · SSM · Jamba
Plain English
An alternative neural network architecture to Transformer attention that processes long sequences in linear time with a fixed-size memory state—avoiding the quadratic slowdown and growing KV cache of standard Transformers.
Technical Overview
Selective State Space Models (Mamba) parameterize continuous-time state transitions as input-dependent recurrences computed via hardware-aware parallel scans on GPUs. Mamba-2 bridged SSMs and linear attention via Structured State Space Duality (SSD), while hybrid architectures like Jamba and Qwen3.5 interleave SSM layers with self-attention and MoE blocks to combine constant-memory long context with sharp in-context recall.
Timeline (3)
2021
- Paper
2023
- Paper
Gu and Dao introduced Mamba, making SSM state-transition parameters input-selective and writing a fused SRAM parallel-scan kernel—achieving 5× higher inference throughput than Transformers at linear O(N) sequence scaling.
2024
- Paper
Dao and Gu published Mamba-2, formalizing Structured State Space Duality (SSD) to unify selective SSMs with linear attention and enabling Tensor Core matrix-multiplication speedups 2–8× faster than Mamba-1.