IndexMapAbout
Model FamilyCross-ModalLLM

Audio & Speech Models

Also known as: Whisper · Speech-to-Speech · Neural TTS · Voice AI

Plain English

Foundation models for understanding and generating human speech, music, and sound—evolving from transcription (Whisper) and text-to-speech into real-time full-duplex voice conversations.

Technical Overview

Modern speech foundation models span encoder-decoder ASR (OpenAI Whisper), neural audio codec language models (VALL-E), and native full-duplex speech-to-speech Transformers (Moshi, GPT-4o Realtime, Gemini Live) that operate directly on discrete audio tokens without intermediate text transcription bottlenecks.

Timeline (3)

RSS

2022

2023

2024

See also (bkz.) (4)

Open full graph