Google documented a data-center accelerator built for neural-network inference. Its matrix unit illustrates how a chip tuned for a particular class of computation can spend more die area on arithmetic.
Custom AI Accelerators
Also known as: AI ASICs · Inference silicon · Purpose-built AI accelerators
Plain English
Custom AI accelerators are chips and systems built around common neural-network operations. They include Google TPUs, AWS Inferentia and Trainium, and wafer-scale designs from Cerebras.
Technical Overview
Specialized systems balance matrix throughput, memory capacity and bandwidth, and inter-chip communication around a particular workload. TPUs use systolic arrays, while wafer-scale systems keep more data movement within one piece of silicon. Performance depends on the compiler and runtime as well as the workload, so benchmarks should distinguish time to first token, generation speed, and sustained throughput.
Timeline (4)
2017
Paper2019
ReleaseAWS launched EC2 Inf1 instances powered by Inferentia. Customers could compile models with the Neuron SDK and rent inference-specific instances as an alternative to GPU instances.
2024
ReleaseCerebras opened an inference API serving 8B and 70B Llama 3.1 models in 16-bit precision on its third-generation Wafer Scale Engine. This made wafer-scale inference available as a hosted API.
2025
ReleaseAWS launched Trn3 UltraServers with up to 144 Trainium3 chips connected by NeuronLink. The scale-up system lets training workloads distribute model state across a shared high-bandwidth interconnect.
See also (6)
Open full graphCustom AI Accelerators