Foundation Model Ecosystem Map
40 subjects across 6 architectural & ecosystem layers connected by 317 cross-links and typed relations.
Core Architectures & Math
Foundational neural building blocks for representing, attending to, and generating text, images, and video.
Diffusion Transformer (DiT)
An architecture that swaps out the older convolutional U-Net inside image and video generators for a Transformer—bringing the same massive scaling laws that powered LLMs into visual generation.
Diffusion Models
The dominant family of generative models for images and video. They learn to create realistic visuals by starting from pure random noise and gradually sculpting it into a coherent picture over multiple steps.
Transformer
The core neural network architecture behind modern LLMs and increasingly image and video generators. Instead of reading data strictly step-by-step like older recurrent networks, a Transformer compares every piece of an input sequence against every other piece in parallel.
KV Cache
A short-term memory buffer stored in GPU VRAM while an LLM generates text. Without it, the model would have to re-read the entire conversation from scratch for every single new word it outputs.
Multimodal & Vision-Language Models
AI models that understand—and increasingly generate—multiple types of media together, such as text, images, screenshots, audio, and video, inside one shared model.
Latent Space & VAE
A learned compression layer that shrinks high-resolution images or videos into a compact numerical map (latent space) so generative models can work faster, then expands the result back into crisp pixels.
Tokenization
How AI models chop text (and increasingly audio or image patches) into numerical chunks called tokens before processing them. A token is roughly three-quarters of an English word, which is also how LLM APIs measure pricing and speed.
Mixture of Experts (MoE)
A design that packs multiple specialized sub-networks ('experts') inside one large model, but only activates a small fraction of them for any given token—delivering big-model intelligence at small-model speed.
Weights, Formats & Adaptation
How model parameters are packaged, quantized, fine-tuned, aligned, and safely distributed.
Checkpoints
A saved snapshot of a neural network's learned numbers (weights) at a specific point in training. Downloading a 'checkpoint' gives you the complete, runnable brain of a model—whether an LLM or an image generator.
LoRA & QLoRA
A lightweight 'patch' file (often tens of megabytes instead of gigabytes) that teaches an existing model a new style, character, domain, or skill without retraining or duplicating the entire base model.
Quantization
Shrinking a model's weights from high-precision decimal numbers (like 16-bit floats) down to smaller numbers (8-bit or 4-bit) so the model uses far less GPU memory and runs faster, with minimal loss in quality.
Safetensors
The modern, safe file format for storing and sharing AI model weights. It replaced older Python pickle files (.ckpt/.bin) that could hide malicious code, and it loads models into GPU memory much faster.
GGUF
A single-file model format built for running AI models locally on laptops, Apple Silicon Macs, and consumer GPUs—packaging compressed weights and tokenizer metadata together so you can run a model with zero complex setup.
Distillation & Synthetic Data
Using a huge, expensive 'teacher' model to generate high-quality training examples or supervision signals so a much smaller, faster, cheaper 'student' model can punch far above its weight.
Post-Training (SFT, RLHF, DPO & GRPO)
Everything done to a model *after* initial pre-training to turn a raw text-autocomplete engine into a helpful, safe assistant or a multi-step reasoning model that checks its own work.
ControlNet & IP-Adapter
Add-on modules that give you precise visual steering over an image or video generator—letting you lock in a pose, architectural layout, depth map, sketch, or reference face/style instead of relying on text prompts alone.
Inference, Serving & Controls
Systems engineering, GPU kernels, memory optimization, and sampling controls for running models fast.
Inference Engines (vLLM, SGLang, llama.cpp)
Specialized runtime servers that actually execute AI models in production or on your laptop—squeezing 5–20× more speed and concurrent users out of the same GPU compared to naive Python scripts.
Context Window & RoPE
The working memory limit of a model—how many tokens of text, code, or video frames it can 'see' at once—together with RoPE, the math trick that lets models know which token came before which even across hundreds of thousands of tokens.
Classifier-Free Guidance (CFG)
The 'prompt adherence slider' in image and video generators. Turning CFG up forces the model to follow your text prompt more strictly—and enables negative prompts—while turning it too high oversaturates or 'burns' the image.
FlashAttention & GPU Kernels
Low-level GPU engineering breakthroughs that compute Transformer attention 2–8× faster and with far less memory—without changing the mathematical result—by keeping data inside ultra-fast on-chip SRAM instead of constantly shuttling it to main VRAM.
Sampling & Schedulers
The math routines that turn raw model predictions into actual outputs—controlling creativity vs. determinism in LLMs (temperature, top-p) and controlling how noise is stepped down into pixels in image and video models (Euler, DPM++, FlowMatch).
Speculative Decoding
A lossless speedup trick for LLMs: a tiny fast 'draft' model guesses the next several words ahead, and the big smart model verifies all of them in a single parallel check—generating 2–3× faster with mathematically identical output quality.
Labs, Companies & Hubs
Research labs, open-weight ecosystems, and distribution platforms shaping the foundation model landscape.
Hugging Face
The 'GitHub of AI'—the central open platform where researchers, labs, and developers host, discover, and download open-weight model checkpoints, datasets, demos (Spaces), and core Python libraries.
OpenAI
The AI research lab behind ChatGPT, the GPT model lineage, reasoning models (o-series), and the Sora video generator. Its API design also became the unofficial HTTP standard copied by almost every open-source serving engine.
DeepSeek
A Chinese AI research lab that shook the global AI industry by releasing MIT-licensed open-weight models matching top proprietary labs at a fraction of reported training and inference costs, while publishing full architectural blueprints.
Black Forest Labs
The German AI lab founded by the original creators of Latent Diffusion / Stable Diffusion. They build the FLUX family of image models, which quickly became the gold standard for open-weight photorealism and typography.
Google DeepMind
Google's flagship AI research organization—home to the Gemini multimodal LLMs, the open-weight Gemma family, Imagen image models, Veo video models, and foundational breakthroughs like the original Transformer and AlphaFold.
Anthropic
The AI safety and research lab behind the Claude family of models, known for strong software engineering performance, long-context reasoning, mechanistic interpretability research, and creating the Model Context Protocol (MCP).
Meta AI
Meta's AI research and product group (including FAIR), famous for kickstarting the open-weight LLM ecosystem with the Llama family as well as foundational open-source tools like PyTorch and Segment Anything (SAM).
Runway
One of the earliest and most influential AI video labs—co-creator of the original Stable Diffusion research and builder of the Gen-2, Gen-3, and Gen-4 video generation models geared toward filmmakers and studios.
Representative Model Families
Landmark open-weight and frontier model lineages across text, image, and video generation.
FLUX
A family of open-weight and API image generation models from Black Forest Labs that replaced Stable Diffusion XL as the community's go-to base model for sharp photorealism, accurate text in images, and custom LoRA training.
Video Generation Models (Sora, Veo, Kling, Wan)
AI systems that synthesize multi-second video clips from text prompts, reference images, or camera controls—combining visual realism, motion physics, and increasingly synchronized audio.
Llama
Meta's flagship family of open-weight language and multimodal models. For years, Llama served as the reference architecture around which almost every open-source LLM runtime, fine-tuning tool, and quantization format was built.
Workflows, Tooling & Agent Stack
Node editors, model marketplaces, inference clouds, retrieval pipelines, protocols, and evaluation harnesses.
ComfyUI
The dominant open-source node-graph studio for image and video AI. Instead of a fixed form with a few sliders, ComfyUI lets you wire together checkpoints, LoRAs, ControlNets, VAEs, and upscalers like modular audio synths—and export the exact graph as JSON.
Serverless GPU & Inference Clouds
Pay-per-use cloud platforms that host open-weight LLMs, image/video models, and custom GPU code behind simple HTTP APIs—so developers don't have to rent idle GPUs or manage Kubernetes GPU clusters themselves.
Civitai
The largest community hub and gallery specifically for open-source image and video AI assets—where creators share fine-tuned checkpoints, style/character LoRAs, ControlNets, and ComfyUI workflows alongside visual preview examples.
Evals & Benchmarks
The test suites and leaderboards that measure whether an AI model—or an AI application you built—is actually getting better or silently regressing across accuracy, coding, visual realism, safety, and latency.
RAG & Hybrid Retrieval
Giving an AI model a live search engine over your own documents, databases, or code before it answers—so responses stay grounded in up-to-date, verifiable facts instead of relying only on frozen training weights.
Model Context Protocol (MCP)
An open 'USB-C standard' for plugging AI models and agents into external tools, databases, APIs, and local files—so developers write a connector once instead of rebuilding custom integrations for every AI app.
LangGraph & Agent Orchestration
A framework for building reliable, multi-step AI workflows as explicit state machines (graphs)—supporting loops, retries, human-in-the-loop approvals, and durable memory rather than fragile one-shot prompts.