Wortsman et al. published Model Soups, showing that averaging the weights of multiple fine-tuned runs from the same pre-trained checkpoint improves accuracy and robustness with zero extra inference cost.
Model Merging
Also known as: Weight Merging · SLERP · TIES-Merging · MergeKit
Plain English
Combining the weights of two or more fine-tuned AI models into a single checkpoint without running any new training—blending skills like coding, math, and chat for free.
Technical Overview
Model merging operates directly in parameter space on checkpoints sharing a common base initialization—using linear averaging (Model Soups), spherical interpolation (SLERP), task-vector arithmetic, or sign-conflict resolution (TIES-Merging) and random drop-and-rescale (DARE). Orchestrated by tools like MergeKit, merging is widely used to combine specialized LoRA or full-weight fine-tunes across open-weight leaderboards.
Timeline (4)
2022
- Paper
- Paper
Ilharco et al. introduced Task Arithmetic, defining task vectors (
θ_ft - θ_base) that can be added to compose capabilities or subtracted to unlearn unwanted behaviors without gradient updates.
2023
- Paper
Yadav et al. proposed TIES-Merging, resolving sign conflicts and redundant parameter magnitudes across task vectors before averaging—preventing destructive interference when combining three or more specialized Checkpoints.
2024
- Paper
Goddard et al. published MergeKit, an open-source toolkit implementing SLERP, TIES, DARE, and Frankenmerging that powered a wave of community-merged open-weight models on Hugging Face.