Ai2 and University of Washington researchers (Matthew E. Peters et al.) published Deep contextualized word representations (ELMo), deriving token Embeddings from the internal states of a deep bidirectional LSTM language model so a word's vector shifts dynamically with its surrounding sentence context—establishing the pretrain-then-adapt paradigm shortly before BERT.
Allen Institute for AI
Also known as: Ai2 · AllenAI · OLMo · Molmo · Tülu
Plain English
The Seattle-based nonprofit research institute founded by Paul G. Allen in 2014 that builds fully open language, multimodal, and reasoning models—publishing not only model weights (OLMo, Molmo, Tülu) but also complete pretraining corpora (Dolma), training code, intermediate checkpoints, and reinforcement learning recipes.
Technical Overview
Ai2 pioneered contextual word representations (ELMo) and today anchors the scientific open-model ecosystem: unlike open-weight releases that share only final parameters, Ai2 ships end-to-end artifacts under permissive Apache-2.0 licenses, including the multi-trillion-token Dolma pretraining corpora, dense and sparse MoE Olmo and OLMoE Checkpoints, the Tülu Post-Training (SFT, RLHF, DPO & GRPO) pipeline (SFT, DPO, and Reinforcement Learning with Verifiable Rewards), the Molmo vision-language models for image and video understanding, and the Olmo-core Distributed Training stack.
Timeline (6)
2018
Paper2024
ReleaseAi2 launched OLMo (1B and 7B) alongside the 3-trillion-token Dolma open corpus, training code, logs, and hundreds of intermediate Pre-Training Checkpoints under Apache 2.0, enabling researchers to audit data attribution, training dynamics, and memorization across the full pretraining lifecycle.
- Paper
Following the September 3 release of OLMoE (a 7B-total, 1B-active open Mixture-of-Experts language model), Ai2 introduced Molmo and the PixMo dataset, training open vision-language models (1B–72B) from scratch on human speech-transcribed image descriptions and 2D pixel-coordinate pointing data without distilling from proprietary APIs.
- Paper
Ai2 published the Tülu 3 post-training recipe, combining supervised fine-tuning and direct preference optimization with reinforcement learning from verifiable rewards. For tasks with checkable answers or instructions, the recipe could derive rewards from a verifier.
2025
ReleaseAi2 released the 7B and 32B Olmo 3 family with intermediate checkpoints, data, and recipes for successive training stages. The artifacts let researchers inspect how base models became instruction-tuned or reasoning models.
2026
ReleaseAi2 released Olmo-core 3, an open training framework for large mixtures of experts. Expert parallelism spreads expert weights across devices so the model can grow beyond the memory of a single device.
See also (17)
Open full graphAllen Institute for AI
12 more connections
- Multimodal & Vision-Language ModelsOutgoing · Enables
- Open-Weights LicensingOutgoing · Enables
- Olmo (Open Language Model)Outgoing · MentionsIncoming · Created by
- BERTOutgoing · Mentions
- CheckpointsOutgoing · Mentions
- Distributed TrainingOutgoing · Mentions
- DPOOutgoing · Mentions
- EmbeddingsOutgoing · Mentions
- GRPOOutgoing · Mentions
- Mixture of Experts (MoE)Outgoing · Mentions
- Recurrent Neural NetworksOutgoing · Mentions
- Supervised Fine-Tuning (SFT)Outgoing · Mentions