Wang et al. at Microsoft published VALL-E, reframing text-to-speech as discrete token prediction over EnCodec neural audio codec representations and proving that a language-model architecture can clone an unseen speaker's voice zero-shot from a 3-second acoustic prompt.
Text-to-Speech (TTS) & Voice Cloning
Also known as: TTS · Neural TTS · Voice Cloning · F5-TTS · Kokoro-82M
Plain English
AI models that convert written text into natural-sounding human speech, capable of matching a reference speaker's voice, emotion, and pacing from just a few seconds of audio.
Technical Overview
Modern neural TTS has converged on two main paradigms: autoregressive or masked Transformers predicting discrete Residual Vector Quantization (RVQ) neural audio codec tokens (VALL-E, Sesame CSM), and non-autoregressive flow-matching models (F5-TTS) or compact adversarial acoustic decoders (StyleTTS 2, Kokoro-82M) that synthesize mel-spectrograms or waveforms in parallel.
Timeline (4)
2023
Paper2024
PaperYushen Chen et al. published F5-TTS (A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching), pairing a ConvNeXt text refiner with a non-autoregressive Diffusion Transformer trained via flow matching to achieve fast, zero-shot open-weight voice cloning without duration predictors or phoneme alignment.
- Release
Developer
hexgradreleased Kokoro-82M under Apache-2.0 on Hugging Face, demonstrating that an 82-million-parameter StyleTTS 2 / ISTFTNet architecture trained on curated synthetic audio can rival multi-billion-parameter TTS models while running faster than real time on consumer CPUs. 2025
PaperSesame published Crossing the Uncanny Valley of Voice and subsequently open-sourced CSM-1B (Conversational Speech Model), conditioning a dual-autoregressive Transformer over interleaved text and Mimi RVQ audio tokens on prior conversation history so prosody, pauses, and breath adapt to dialogue context.
See also (8)
Open full graphText-to-Speech (TTS) & Voice Cloning