Devlin et al. at Google AI Language published BERT, showing that pre-training a deep bidirectional Transformer encoder via Masked Language Modeling produced a single backbone that swept 11 NLP benchmarks after lightweight fine-tuning.
BERT
Also known as: RoBERTa · ModernBERT · Bidirectional Encoder
Plain English
Google's bidirectional Transformer architecture that reads text from both left and right at once—powering search ranking, embeddings, rerankers, and classification rather than chat generation.
Technical Overview
Pre-trained with Masked Language Modeling (MLM) rather than causal next-token prediction, BERT encodes full-sequence bidirectional context into dense token and sequence representations. While decoder-only GPT models took over open-ended generation, encoder-only BERT descendants (RoBERTa, DeBERTa, ModernBERT) remain the workhorses for embeddings, cross-encoder reranking in RAG, and low-latency classification.
Timeline (3)
2018
- Paper
2019
2024
- Paper
Warner et al. released ModernBERT, updating the encoder-only Transformer with RoPE, GeGLU layers, alternating local/global attention, unpadding, and an 8,192-token context window trained on 2T tokens for modern embedding and RAG reranking pipelines.
See also (bkz.) (8)
Open full graphBERT