Sepp Hochreiter and Jürgen Schmidhuber published Long Short-Term Memory (LSTM), adding input, forget, and output gates around a constant error carousel so recurrent networks could retain information across hundreds of steps without vanishing gradients.
Recurrent Neural Networks
Also known as: RNN · LSTM · Long Short-Term Memory · Seq2Seq
Plain English
Sequence models that read text or time-series data one token at a time, passing an internal memory vector forward from each step to the next like a running summary.
Technical Overview
Maintain a recurrent hidden state h_t = f(h_{t-1}, x_t) updated sequentially across timesteps. Gated variants like LSTM mitigated vanishing gradients over longer sequences, and encoder-decoder Seq2Seq architectures introduced Attention before the Transformer replaced sequential recurrence with parallelizable self-attention.
Timeline (3)
1997
Paper2014
PaperIlya Sutskever, Oriol Vinyals, and Quoc Le introduced Sequence-to-Sequence (Seq2Seq) learning with multi-layer LSTMs, showing that an encoder-decoder neural network could map variable-length input sentences directly to variable-length translations.
2017
MilestoneVaswani et al.'s Transformer (Attention Is All You Need) replaced step-by-step recurrent loops with parallelizable self-attention, eliminating the sequential training bottleneck that limited RNN and LSTM scaling.