You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/hub/dduf.md
+21-18Lines changed: 21 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,13 +12,13 @@ DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format design
12
12
13
13
This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format.
14
14
15
-
We've seeded some DDUFs of popular formats for the community to play with: https://huggingface.co/DDUF, check them out!
15
+
Check out the [DDUF](https://huggingface.co/DDUF) org to start using some of the most popular diffusion models in DDUF.
16
16
17
17
<Tip>
18
18
19
19
We welcome contributions with open arms!
20
20
21
-
To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know! Discussions about the DDUF format happen in the [DDUF organization](https://huggingface.co/spaces/DDUF/README/discussions/2).
21
+
To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us knowin the DDUF organization[discussions](https://huggingface.co/spaces/DDUF/README/discussions/2).
22
22
23
23
</Tip>
24
24
@@ -30,14 +30,14 @@ The primary goal of DDUF is to create a community-endorsed single-file format fo
30
30
31
31
The DDUF format is also designed to be language-agnostic. While we currently provide tooling for the Python ecosystem, there's nothing stopping similar tools from being developed in JavaScript, Rust, C++, and other languages. Like GGUF or safetensors, DDUF is built to be parsable from a remote location without downloading the entire file, which will enable advanced support on the Hugging Face Hub.
32
32
33
-
## Key Features
33
+
Its key features include the following.
34
34
35
35
1.**Single file** packaging.
36
36
2. Based on **ZIP file format** to leverage existing tooling.
37
37
3. No compression, ensuring **`mmap` compatibility** for fast loading and saving.
38
38
4.**HTTP-friendly**: metadata and file structure can be fetched remotely using HTTP Range requests.
39
-
5.**Flexible**: each model component is stored in its own directory, following the current `diffusers` structure.
40
-
6.**Safe**: uses `safetensors` as weights saving format and prohibits nested directories to prevent ZIP-bombs.
39
+
5.**Flexible**: each model component is stored in its own directory, following the current Diffusers structure.
40
+
6.**Safe**: uses [Safetensors](https://huggingface.co/docs/diffusers/using-diffusers/other-formats#safetensors) as a weight-saving format and prohibits nested directories to prevent ZIPbombs.
41
41
42
42
## Technical specifications
43
43
@@ -59,7 +59,7 @@ The `huggingface_hub` provides tooling to handle DDUF files in Python. It includ
59
59
60
60
### How to read a DDUF file?
61
61
62
-
Reading a DDUF file is as simple as calling `read_dduf_file`and passing a path as argument. Only the metadata is read, meaning this is a lightweight call that will not make your memory explode. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally.
62
+
Pass a path to `read_dduf_file`to read a DDUF file. Only the metadata is read, meaning this is a lightweight call that won't explode your memory. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally.
63
63
64
64
```python
65
65
>>>from huggingface_hub import read_dduf_file
@@ -68,7 +68,7 @@ Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path
This will return a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead.
71
+
`read_dduf_file` returns a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset, and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead.
72
72
73
73
For instance, here is how to load the `model_index.json` content:
74
74
```python
@@ -85,11 +85,12 @@ For binary files, you'll want to access the raw bytes using `as_mmap`. This retu
85
85
... state_dict = safetensors.torch.load(mm) # `mm` is a bytes object
86
86
```
87
87
88
-
Note: `as_mmap` must be used in a context manager to benefit from the memory-mapping properties.
88
+
> [!TIP]
89
+
>`as_mmap` must be used in a context manager to benefit from the memory-mapping properties.
89
90
90
91
### How to write a DDUF file?
91
92
92
-
A DDUFfile can be exported by passing to `export_folder_as_dduf`a folder path containing a diffusion model:
93
+
Pass a folder path to `export_folder_as_dduf`to export a DDUFfile.
93
94
94
95
```python
95
96
# Export a folder as a DDUF file
@@ -99,7 +100,7 @@ A DDUF file can be exported by passing to `export_folder_as_dduf` a folder path
99
100
100
101
This tool scans the folder, adds the relevant entries and ensures the exported fileis valid. If anything goes wrong during the process, a `DDUFExportError`is raised.
101
102
102
-
For more flexibility, you can use [`export_entries_as_dduf`] andpass a list of files to include in the final DDUFfile:
103
+
For more flexibility, use [`export_entries_as_dduf`] to explicitly specify a list of files to include in the final DDUFfile:
103
104
104
105
```python
105
106
# Export specific files from the local disk.
@@ -117,7 +118,7 @@ For more flexibility, you can use [`export_entries_as_dduf`] and pass a list of
117
118
... )
118
119
```
119
120
120
-
This works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively:
121
+
`export_entries_as_dduf` works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively:
121
122
122
123
```python
123
124
(...)
@@ -146,21 +147,23 @@ ZIP provides several advantages:
146
147
- Built-infile indexing
147
148
- Wide language support
148
149
149
-
Why not use a TARwith a table of contents at the beginning of the archive? See explanation [in this comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726).
150
+
### Why not use a TAR with a table of contents at the beginning of the archive?
151
+
152
+
See the explanation in this [comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726).
150
153
151
154
### Why no compression?
152
155
153
-
- Enables direct memory mapping of large files.
154
-
- Ensures consistent and predictable remote file access.
155
-
- Prevents CPU overhead during file reading.
156
-
- Maintains compatibility with safetensors.
156
+
- Enables direct memory mapping of large files
157
+
- Ensures consistent and predictable remote file access
158
+
- Prevents CPU overhead during file reading
159
+
- Maintains compatibility with safetensors
157
160
158
161
### Can I modify a DDUF file?
159
162
160
163
No. For now, DDUF files are designed to be immutable. To update a model, create a new DDUFfile.
We are continuously reaching out to other libraries and frameworks, if you are interested in adding support for your project, open a Discussion in the [DDUF org](https://huggingface.co/spaces/DDUF/README/discussions).
169
+
We are constantly reaching out to other libraries and frameworks. If you are interested in adding support to your project, open a Discussion in the [DDUF org](https://huggingface.co/spaces/DDUF/README/discussions).
0 commit comments