Wenyi Hong et al. at Tsinghua University and BAAI published CogVideo, a 9B-parameter autoregressive text-to-video Transformer trained on top of the frozen CogView2 text-to-image model using multi-frame-rate hierarchical frame interpolation.
CogVideo & CogVideoX
Also known as: CogVideo · CogVideoX · CogVideoX-5B
Plain English
Tsinghua KEG and Zhipu AI's family of open-weight video generation models that pioneered full 3D spatiotemporal attention and 3D causal VAEs in the open-source video ecosystem.
Technical Overview
While early video models stacked separate 1D temporal attention layers onto 2D image networks, CogVideoX paired a 3D causal VAE (4x8x8 compression) with an Expert Diffusion Transformer using full 3D attention and Expert Adaptive LayerNorm to align T5 text embeddings with spatiotemporal video tokens.
Timeline (4)
2022
Paper2024
ReleaseZhipu AI and Tsinghua open-sourced CogVideoX-2B alongside its technical report (CogVideoX: Text-to-Video Diffusion Models with an Expert Transformer), replacing separable spatial-temporal attention with full 3D spatiotemporal attention over a 3D causal VAE and shipping day-zero integration in Hugging Face Diffusers and ComfyUI.
- Release
The CogVideo team released CogVideoX-5B (followed by
CogVideoX-5B-I2Vfor image-to-video in September 2024), scaling the Expert DiT backbone to 5 billion parameters and establishing the primary open-weight video LoRA fine-tuning target of late 2024. - Release
Zhipu AI and Tsinghua released CogVideoX1.5-5B (
T2VandI2V), upgrading the open-weight family to 10-second clips at 768p–1360p resolution with arbitrary aspect-ratio support.
See also (11)
Open full graphCogVideo & CogVideoX