IndexMapAbout
ConceptLLMImageCross-Modal

Model Merging

Also known as: Weight Merging · SLERP · TIES-Merging · MergeKit

Plain English

Combining the weights of two or more fine-tuned AI models into a single checkpoint without running any new training—blending skills like coding, math, and chat for free.

Technical Overview

Model merging operates directly in parameter space on checkpoints sharing a common base initialization—using linear averaging (Model Soups), spherical interpolation (SLERP), task-vector arithmetic, or sign-conflict resolution (TIES-Merging) and random drop-and-rescale (DARE). Orchestrated by tools like MergeKit, merging is widely used to combine specialized LoRA or full-weight fine-tunes across open-weight leaderboards.

Timeline (4)

RSS

2022

  1. Paper

    Ilharco et al. introduced Task Arithmetic, defining task vectors (θ_ft - θ_base) that can be added to compose capabilities or subtracted to unlearn unwanted behaviors without gradient updates.

2023

2024

See also (bkz.) (3)

Open full graph