Google DeepMind described RT-2, using a shared token format for text and robot actions. Training on web vision-language data together with robot trajectories helped the policy use visual concepts beyond its manipulation demonstrations.
Vision-Language-Action Models
Also known as: VLA · Robot foundation models · OpenVLA · RT-2 · π0
Plain English
A vision-language-action model uses images and instructions to decide what a robot should do next. Its outputs can control a robot’s movement or manipulation of objects.
Technical Overview
A VLA extends a vision-language backbone with an action representation. The policy can predict discretized action tokens or generate multiple continuous control steps together with methods such as flow matching. Results depend on the robot hardware, the demonstrations used for training, and how the policy is evaluated on unfamiliar tasks or environments.
Timeline (4)
2023
Paper2024
PaperResearchers presented OpenVLA, a 7B-parameter model trained on demonstrations from multiple robot platforms. With released weights and fine-tuning code, researchers could adapt the same starting checkpoint to a different setup.
- Paper
Physical Intelligence described π0, which uses a vision-language backbone and a flow-matching action head to generate chunks of continuous robot actions. The same policy could be applied to tasks involving different robot designs.
2026
PaperPhysical Intelligence described π0.7, which conditions actions on optional subgoal images and task metadata as well as language. Its experiments included multi-stage manipulation in unfamiliar environments.