The Vidur authors profiled model operations on hardware and used those measurements to simulate end-to-end latency and throughput. This let them compare batching policies and parallelism choices for an inference engine before paying to run each full deployment.
LLM Serving Benchmarks
Also known as: Inference benchmarking · Serving performance
Plain English
A serving benchmark sends a defined workload to a model server and measures what users experience as traffic grows. A server can generate many tokens per second overall while an individual user still waits a long time for a reply to start.
Technical Overview
Time to first token includes queueing and prompt processing. Time per output token measures generation after the first token, while aggregate throughput counts work across requests. Comparing inference engines requires reporting the workload, including request arrival rate and input and output lengths, alongside these measurements.
Chronological Entries (2)
MLCommons added Llama 2 70B to MLPerf Inference v4.0. Its server scenario tested randomly arriving requests with a two-second limit for the first token and a 200-millisecond limit for time per subsequent output token. That made both the initial wait and the pace of the response part of the test.
Direct Relations (1)
- Part ofEvals & Benchmarks