IndexMapAbout
ArchitectureLLM

BERT

Also known as: RoBERTa · ModernBERT · Bidirectional Encoder

Plain English

Google's bidirectional Transformer architecture that reads text from both left and right at once—powering search ranking, embeddings, rerankers, and classification rather than chat generation.

Technical Overview

Pre-trained with Masked Language Modeling (MLM) rather than causal next-token prediction, BERT encodes full-sequence bidirectional context into dense token and sequence representations. While decoder-only GPT models took over open-ended generation, encoder-only BERT descendants (RoBERTa, DeBERTa, ModernBERT) remain the workhorses for embeddings, cross-encoder reranking in RAG, and low-latency classification.

Timeline (3)

RSS

2018

2019

2024

See also (bkz.) (8)

Open full graph
3 more connections