The Sarathi-Serve authors split incoming prompts into chunks and scheduled those chunks alongside ongoing decode steps. This lets an inference engine start work on a long prompt while still emitting tokens for earlier requests.
Continuous Batching
Also known as: Iteration-level scheduling · In-flight batching
Plain English
A language model server can change which requests it processes together after each generation step. When one reply finishes, another prompt can join while longer replies keep going.
Technical Overview
The scheduler in an inference engine forms a batch for each iteration and retains the KV cache of each unfinished request. Processing a long new prompt in one iteration can delay tokens for users already receiving a reply. Chunked prefill lets the scheduler limit how much prompt work it admits per iteration.
Chronological Entries (2)
At OSDI, Yu and colleagues presented Orca, a serving system that schedules Transformer requests one model iteration at a time. A finished request can return its reply and free a slot for a waiting request while the rest of the batch continues generating.