I attempted to document the internals of ComfyUI's engine for modders/developers

[horsten]

I've often found it tricky to navigate the both brilliantly efficient but also very undocumented internals of ComfyUI when writing custom nodes, patching models, or generally hacking around. There are a lot of implicit "API contracts" and core concepts that are non-obvious because they're not documented anywhere, so it often involves trying to find a "close enough example" of what you want to do if you're not a core developer and trying to understand the system.

I've tried to document the key concepts that one needs to understand when adding features, models, or experiments. It started mostly for my own benefit, but I hope others might find it useful - perhaps especially those who create "wrapper nodes" for external models and similar, often resulting in something very fragile that doesn't coexist happily with anything else Comfy.

Below's the README, and the repository can be found here: https://github.com/horsten/comfyui-internals-doc

Feedback and pull requests are welcome.

ComfyUI Internals: The Missing Manual

"The missing blueprint for the ComfyUI Engine."

Overview

ComfyUI has become the de-facto operating system for open-source generative AI. However, its core architecture—the execution engine, memory management, and sampling loop—has remained largely undocumented, residing only in the source code and the minds of its original creators and the core development team. The code and design of these core parts is brilliant, complex, and highly optimized, which is why ComfyUI works so well, but for the same reasons it's very easy to break it when you want to add something or change some behavior in your specific use case.

This repository aims to solve the "Black Box" problem.

It provides deep architectural analysis, explicit API contracts, and reference documentation for developers who want to write robust custom nodes, implement new model architectures, or simply understand how the engine works under the hood.

🛑 Why This Matters: The "Ghost" Bugs

Without understanding these internals, developers often encounter "Ghost Bugs"—issues that appear randomly or only on specific hardware. This documentation explains the root causes of:

• The "Wrapper Node" Anti-Pattern: A common mistake where developers instantiate a model inside a node (e.g., self.pipeline = DiffusersPipeline(...)).
  • The Crash: This creates "Dark Matter" in VRAM that the ModelManagement system cannot see or offload, causing immediate OOMs when other nodes try to run. It also prevents your "wrapper" from taking advantage of the very features that makes ComfyUI so fast, and efficient, often negating the impact of your custom model "optimizations".
  • The Fix: Always wrap models in ModelPatcher and return them. Let the engine handle the loading. People who write wrapper nodes often add inferior versions of features that are already in the core engine, but finding what you're looking for, and understanding how to use it, can be incredibly complex if you have to deduce it all by finding a similar "example" somewhere in the code.
• The "Black Image" Output: Often caused by VAE decoding on the wrong device or dtype mismatches during VRAM offloading.
• Random OOMs (Out of Memory): Caused by manually moving tensors to CUDA (.to('cuda')) instead of using ModelPatcher, which bypasses the memory manager's accounting.
• Garbage/Static Noise Generation: A result of the NHWC vs NCHW shape mismatch when using standard PyTorch operations on ComfyUI image tensors.
• Workflow "Bleed": When a node modifies a model in-place (without .clone()), causing side effects to leak into other branches of the workflow.
• Silent Model Loading Failures: When comfy.sd guesses the wrong architecture (e.g., loading an SDXL-Turbo model as SD1.5) because of ambiguous state dict keys.

📚 The Architecture (Textbook)

If you are new to ComfyUI development (and even if you're not, because unless you're a member of the core development team, you probably, like me, had to guess about a great number of things), read these in order to build a mental model of the system.

1. The Object Lifecycle Understanding ModelPatcher, model_management, and why you must always .clone() before modifying.
2. The Execution Engine How the graph is sorted, validated, and executed. Explains Caching, Lazy Evaluation, and Subgraphs.
3. The Node Interface How INPUT_TYPES, RETURN_TYPES, and the V1 vs V3 initialization protocols work.
4. The Sampling Process A deep dive into comfy.samplers. How Guiders, Schedulers, and Conditionings interact to generate an image.
5. Model Loading & Detection Analysis of comfy.sd. How the system "guesses" which model architecture a SafeTensors file contains.
6. Guide: Adding New Architectures A step-by-step guide to implementing support for new Diffusion Models (e.g., Flux, Cosmos, OmniGen).
7. Patching Guide How to inject code into the model (LoRAs, ControlNets, and Attention Hooks) without breaking the engine.

⚡ Quick Reference (Cheatsheet)

For active development, keep these open.

• API Contracts & Pitfalls The "Rules of the Road". Do's and Don'ts to prevent memory leaks and execution bugs.
• Tensor Management Where to create tensors (CPU vs GPU), correct data types, and how to handle intermediate_device.
• Type Definitions What exactly is inside a MODEL, LATENT, or MASK object?

Core Concepts at a Glance

Concept	Description
ModelPatcher	The mutable wrapper around the immutable model weights. Handles LoRAs and VRAM placement.
Intermediate Device	An abstraction that is the GPU on high-end systems, but CPU on low-VRAM systems.
NHWC	ComfyUI uses "Channels Last" (Batch, Height, Width, Channel) format for Images, unlike standard PyTorch (NCHW).
Guider	The component that handles CFG (Classifier Free Guidance) math. It wraps the model during sampling.
Key Sniffing	The method ComfyUI uses to detect model types by looking for specific key names in the state dict.

Contributing

This documentation was generated through a collaboration between human insight (and practical experience of "getting things wrong" and learning where the friction points were), and AI analysis of the source code (Jan 2026). As the ComfyUI codebase evolves, these documents may need updates. Pull Requests to keep this "Source of Truth" accurate are welcome.

I'll try my best to keep it up-to-date with major changes in the core of ComfyUI, especially if I'm notified in the form of a pull requests, and of course if the ComfyUI team would like to "adopt" this documentation and make it part of the main project, I'd welcome that as well.