Skip to content

Commit 7466f78

Browse files
WauplinWauplin
committed
Start DDUF documentation
1 parent b27277a commit 7466f78

File tree

2 files changed

+154
-0
lines changed

2 files changed

+154
-0
lines changed

‎docs/hub/_toctree.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,8 @@
137137
title: Integrate a library with the Hub
138138
- local: models-tasks
139139
title: Tasks
140+
- local: dduf
141+
title: DDUF
140142
- local: gguf
141143
title: GGUF
142144
sections:

‎docs/hub/dduf.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# DDUF: Diffusion ??? Universal Format
2+
3+
## Overview
4+
5+
DDUF (Diffusion ??? Universal Format) is a file format designed to make storing, distributing, and using diffusion models much easier. Built on the ZIP file format, DDUF offers a standardized, efficient, and flexible way to package all parts of a diffusion model into a single, easy-to-manage file.
6+
7+
This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format.
8+
9+
<Tip>
10+
11+
We welcome contributions with open arms!
12+
13+
To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know! Discussions about the DDUF format happen in the https://huggingface.co/DDUF organization.
14+
15+
</Tip>
16+
17+
## Motivation
18+
19+
Yet, another file format? Yes, but for good reasons!
20+
21+
The primary goal of DDUF is to create a community-endorsed file format for diffusion models. Current model distribution methods often involve multiple separate files, different weight-saving formats, and managing files from various locations. DDUF aims to solve these challenges by packaging all model components into a single file, enforcing a consistent structure while being opinionated about saving formats.
22+
23+
The DDUF format is also designed to be language-agnostic. While we currently provide tooling for the Python ecosystem, there's nothing stopping similar tools from being developed in JavaScript, Rust, C++, and other languages. Like GGUF or safetensors, DDUF is built to be parsable from a remote location without downloading the entire file, which will enable advanced support on the Hugging Face Hub.
24+
25+
## Key Features
26+
27+
1. **Single file** packaging.
28+
2. Based on **ZIP file format** to leverage existing tooling.
29+
3. No compression, ensuring **`mmap` compatibility** for fast loading and saving.
30+
4. **HTTP-friendly**: metadata and file structure can be fetched remotely using HTTP Range requests.
31+
5. **Flexible**: each model component is stored in its own directory, following the current `diffusers` structure.
32+
6. **Safe**: uses `safetensors` as weights saving format and prohibits nested directories to prevent ZIP-bombs.
33+
34+
## Technical specifications
35+
36+
Technically, a `.dduf` file **is** a [`.zip` archive](https://en.wikipedia.org/wiki/ZIP_(file_format)). By building on a universally supported file format, we ensure robust tooling already exists. However, some constraints are enforced to meet diffusion models' requirements:
37+
- Data must be stored uncompressed (flag `0`), allowing mmap-compatibility.
38+
- Data must be stored using ZIP64 protocol, enabling saving files above 2GB.
39+
- The archive can only contain `.json`, `.safetensors`, `.model` and `.txt` files.
40+
- A `model_index.json` file must be present at the root of the archive. It must contain a key-value mapping with metadata about the model and its components.
41+
- Each component must be stored in its own directory (e.g., `vae/`, `text_encoder/`). Nested files must use UNIX-style path separators (`/`).
42+
- Each directory must correspond to a component in the `model_index.json` index.
43+
- Each directory must contain a json config file (one of `config.json`, `tokenizer_config.json`, `image_processor.json`).
44+
- Sub-directories are forbidden.
45+
46+
Want to check if your file is valid? Check it out using this Space: https://huggingface.co/spaces/DDUF/dduf-check.
47+
48+
## Usage
49+
50+
The `huggingface_hub` provides tooling to handle DDUF files in Python. It includes built-in rules to validate file integrity and helpers to read and export DDUF files. The goal is to see this tooling adopted in the Python ecosystem, such as in the `diffusers` integration. Similar tooling can be developed for other languages (JavaScript, Rust, C++, etc.).
51+
52+
### How to read a DDUF file?
53+
54+
Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path as argument. Only the metadata is read, meaning this is a lightweight call that will not explode your memory. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally.
55+
56+
```python
57+
>>> from huggingface_hub import read_dduf_file
58+
59+
# Read DDUF metadata
60+
>>> dduf_entries = read_dduf_file("FLUX.1-dev.dduf")
61+
```
62+
63+
This will return a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead.
64+
65+
For instance, here is how to load the `model_index.json` content:
66+
```python
67+
>>> import json
68+
>>> json.loads(dduf_entries["model_index.json"].read_text())
69+
{'_class_name': 'FluxPipeline', '_diffusers_version': '0.32.0.dev0', '_name_or_path': 'black-forest-labs/FLUX.1-dev', ...
70+
```
71+
72+
For binary files, you'll want to access the raw bytes using `as_mmap`. This returns bytes as a memory-mapping on the original file. The memory-mapping allows you to read only the bytes you need without loading everything in memory. For instance, here is how to load safetensors weights:
73+
74+
```python
75+
>>> import safetensors.torch
76+
>>> with dduf_entries["vae/diffusion_pytorch_model.safetensors"].as_mmap() as mm:
77+
... state_dict = safetensors.torch.load(mm) # `mm` is a bytes object
78+
```
79+
80+
Note: `as_mmap` must be used in a context manager to benefit from the memory-mapping properties.
81+
82+
### How to write a DDUF file?
83+
84+
A DDUF file can be exported by passing to `export_folder_as_dduf` a folder path containing a diffusion model:
85+
86+
```python
87+
# Export a folder as a DDUF file
88+
>>> from huggingface_hub import export_folder_as_dduf
89+
>>> export_folder_as_dduf("FLUX.1-dev.dduf", folder_path="path/to/FLUX.1-dev")
90+
```
91+
92+
This tool scans the folder, adds the relevant entries and ensures the exported file is valid. If anything goes wrong during the process, a `DDUFExportError` is raised.
93+
94+
For more flexibility, you can use [`export_entries_as_dduf`] and pass a list of files to include in the final DDUF file:
95+
96+
```python
97+
# Export specific files from the local disk.
98+
>>> from huggingface_hub import export_entries_as_dduf
99+
>>> export_entries_as_dduf(
100+
... dduf_path="stable-diffusion-v1-4-FP16.dduf",
101+
... entries=[ # List entries to add to the DDUF file (here, only FP16 weights)
102+
... ("model_index.json", "path/to/model_index.json"),
103+
... ("vae/config.json", "path/to/vae/config.json"),
104+
... ("vae/diffusion_pytorch_model.fp16.safetensors", "path/to/vae/diffusion_pytorch_model.fp16.safetensors"),
105+
... ("text_encoder/config.json", "path/to/text_encoder/config.json"),
106+
... ("text_encoder/model.fp16.safetensors", "path/to/text_encoder/model.fp16.safetensors"),
107+
... # ... add more entries here
108+
... ]
109+
... )
110+
```
111+
112+
This works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively:
113+
114+
```python
115+
(...)
116+
117+
# Export state_dicts one by one from a loaded pipeline
118+
>>> def as_entries(pipe: DiffusionPipeline) -> Generator[Tuple[str, bytes], None, None]:
119+
... # Build a generator that yields the entries to add to the DDUF file.
120+
... # The first element of the tuple is the filename in the DDUF archive. The second element is the content of the file.
121+
... # Entries will be evaluated lazily when the DDUF file is created (only 1 entry is loaded in memory at a time)
122+
... yield "vae/config.json", pipe.vae.to_json_string().encode()
123+
... yield "vae/diffusion_pytorch_model.safetensors", safetensors.torch.save(pipe.vae.state_dict())
124+
... yield "text_encoder/config.json", pipe.text_encoder.config.to_json_string().encode()
125+
... yield "text_encoder/model.safetensors", safetensors.torch.save(pipe.text_encoder.state_dict())
126+
... # ... add more entries here
127+
128+
>>> export_entries_as_dduf(dduf_path="my-cool-diffusion-model.dduf", entries=as_entries(pipe))
129+
```
130+
131+
## F.A.Q.
132+
133+
### Why build on top of ZIP?
134+
135+
ZIP provides several advantages:
136+
- Universally supported file format
137+
- No additional dependencies for reading
138+
- Built-in file indexing
139+
- Wide language support
140+
141+
Why not use a TAR with a table of contents at the beginning of the archive? See explanation [in this comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726).
142+
143+
### Why no compression?
144+
145+
- Enables direct memory mapping of large files.
146+
- Ensures consistent and predictable remote file access.
147+
- Prevents CPU overhead during file reading.
148+
- Maintains compatibility with safetensors.
149+
150+
### Can I Modify a DDUF file?
151+
152+
No. For now, DDUF files are designed to be immutable. To update a model, create a new DDUF file.

0 commit comments

Comments
 (0)