FIFoundationIndex
IndexMapAbout
ConceptLLM

LLM Serving Benchmarks

Also known as: Inference benchmarking · Serving performance

Plain English

A serving benchmark sends a defined workload to a model server and measures what users experience as traffic grows. A server can generate many tokens per second overall while an individual user still waits a long time for a reply to start.

Technical Overview

Time to first token includes queueing and prompt processing. Time per output token measures generation after the first token, while aggregate throughput counts work across requests. Comparing inference engines requires reporting the workload, including request arrival rate and input and output lengths, alongside these measurements.

Chronological Entries (2)

Direct Relations (1)