DistServe chose the GPU allocation and parallelism for each phase to meet separate latency targets for the first token and for subsequent tokens. Its authors measured the request rate a deployment could sustain while meeting both targets.
Prefill/Decode Disaggregation
Also known as: Disaggregated serving · P/D disaggregation · Phase splitting
Plain English
One group of machines reads the prompt, then hands the request to another group that generates the reply token by token. An operator can add capacity to whichever group is making users wait.
Technical Overview
Prefill computes the prompt's key and value tensors. Moving that state to decode workers lets an operator choose hardware and parallelism separately for each phase. The transfer adds latency that depends on the cache size and the bandwidth between workers.
Chronological Entries (2)
The Splitwise authors measured different hardware demands for prompt processing and token generation, then placed the phases on separate machines. They transferred the cached state over the cluster interconnect so a decode worker could continue where the prompt worker left off.
Direct Relations (1)
- UsesKV Cache