FIFoundationIndex
IndexMapAbout
ConceptCross-ModalLLMImageVideo

LoRA & QLoRA

Also known as: LoRA · QLoRA · Low-Rank Adaptation · PEFT

Plain English

A lightweight 'patch' file (often tens of megabytes instead of gigabytes) that teaches an existing model a new style, character, domain, or skill without retraining or duplicating the entire base model.

Technical Overview

Low-Rank Adaptation freezes base model weights W and injects trainable low-rank decomposition matrices ΔW = A·B (rank r ≪ d) into attention and MLP projections. QLoRA pairs LoRA with 4-bit Quantization of the frozen base model, enabling 70B LLM fine-tuning on a single GPU; in image/video tools like ComfyUI and Civitai, multiple LoRAs can be stacked dynamically onto one checkpoint.

Chronological Entries (3)

#1Paper
Curated

Hu et al. at Microsoft introduced LoRA, demonstrating that weight updates during adaptation have low intrinsic rank. Training only tiny adapter matrices cuts trainable parameters by 1,000–10,000× while merging cleanly back into base weights with zero inference latency overhead.

Direct Relations (4)