Radford et al. at OpenAI published Whisper, an open-weight encoder-decoder Transformer trained on 680,000 hours of weakly supervised multilingual audio that became the industry standard for speech recognition and transcription.
Audio & Speech Models
Also known as: Whisper · Speech-to-Speech · Neural TTS · Voice AI
Plain English
Foundation models for understanding and generating human speech, music, and sound—evolving from transcription (Whisper) and text-to-speech into real-time full-duplex voice conversations.
Technical Overview
Modern speech foundation models span encoder-decoder ASR (OpenAI Whisper), neural audio codec language models (VALL-E), and native full-duplex speech-to-speech Transformers (Moshi, GPT-4o Realtime, Gemini Live) that operate directly on discrete audio tokens without intermediate text transcription bottlenecks.
Timeline (3)
2022
- Paper
2023
- Paper
Wang et al. at Microsoft published VALL-E, framing text-to-speech as conditional neural audio codec language modeling—enabling zero-shot voice cloning from a 3-second acoustic prompt.
2024
- Paper
Défossez et al. at Kyutai released Moshi, an open-weight full-duplex speech-to-speech foundation model using the Mimi neural audio codec to listen and speak simultaneously with ~200ms latency.