FIFoundationIndex
IndexMapAbout
ConceptLLM

Continuous Batching

Also known as: Iteration-level scheduling · In-flight batching

Plain English

A language model server can change which requests it processes together after each generation step. When one reply finishes, another prompt can join while longer replies keep going.

Technical Overview

The scheduler in an inference engine forms a batch for each iteration and retains the KV cache of each unfinished request. Processing a long new prompt in one iteration can delay tokens for users already receiving a reply. Chunked prefill lets the scheduler limit how much prompt work it admits per iteration.

Chronological Entries (2)