Facebook and Microsoft introduced ONNX, an open format for representing neural networks so models could move between frameworks such as PyTorch, Caffe2, and the Microsoft Cognitive Toolkit. Hardware vendors could then optimize once for the shared format.
ONNX
Also known as: Open Neural Network Exchange · ONNX Runtime · ORT
Plain English
A shared file format for neural networks that lets a model trained in one framework run in another. ONNX Runtime is the engine most people use to run those files on CPUs, GPUs, phones, and browsers.
Technical Overview
ONNX stores a model as a serialized computation graph of standardized operators plus weights, so exporters in frameworks such as PyTorch can target one intermediate representation. ONNX Runtime loads the graph, applies graph optimizations and operator fusion, and dispatches to hardware through pluggable execution providers; its GenAI library adds the tokenizer, KV-cache, and sampling loop for LLMs. Unlike Safetensors, which only stores tensors, ONNX also encodes the compute graph, which suits portable deployment but needs operator coverage for new architectures. Quantization such as int4 is applied at export time.
Timeline (7)
2017
Release2018
ReleaseMicrosoft open-sourced ONNX Runtime, a high-performance inference engine for ONNX models on Linux, Windows, and macOS across CPUs and GPUs. It had already been used inside Microsoft products before release.
2019
MilestoneONNX Runtime 1.0.0 was published to PyPI, marking the first stable release of the runtime that later became the default way to deploy ONNX models.
2024
ReleaseONNX Runtime 1.17 added Phi-2 optimizations, float16 and int4 Quantization support for LLMs, CUDA 12 packages, and an official WebGPU execution provider for running models in the browser.
- Milestone
ONNX Runtime announced day-one support for the Phi-3 mini models with optimized ONNX weights and the ONNX Runtime Generate() API, which wraps the tokenizer, KV-cache, and sampling loop so local LLM inference needs only a few calls.
2025
ReleaseONNX Runtime 1.23.0 was published to PyPI. Windows ML, Microsoft's on-device inference framework for Windows, is documented as built on ONNX Runtime with the same ONNX APIs, so the runtime underpins local AI on Windows.
2026
ReleaseONNX Runtime 1.32.0 shipped on PyPI with Python 3.11–3.14 packages for x86-64 and Arm64 Linux and Windows, and for Arm64 macOS. The wheels support running exported ONNX graphs on those platforms.
See also (6)
Open full graphONNX