Skip to content

Commit ab18cc8

Browse files
Wauplinstevhliu
andauthored
Apply suggestions from code review
Co-authored-by: Steven Liu <[email protected]>
1 parent 8099156 commit ab18cc8

File tree

1 file changed

+21
-18
lines changed

1 file changed

+21
-18
lines changed

docs/hub/dduf.md

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,13 @@ DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format design
1212

1313
This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format.
1414

15-
We've seeded some DDUFs of popular formats for the community to play with: https://huggingface.co/DDUF, check them out!
15+
Check out the [DDUF](https://huggingface.co/DDUF) org to start using some of the most popular diffusion models in DDUF.
1616

1717
<Tip>
1818

1919
We welcome contributions with open arms!
2020

21-
To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know! Discussions about the DDUF format happen in the [DDUF organization](https://huggingface.co/spaces/DDUF/README/discussions/2).
21+
To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know in the DDUF organization [discussions](https://huggingface.co/spaces/DDUF/README/discussions/2).
2222

2323
</Tip>
2424

@@ -30,14 +30,14 @@ The primary goal of DDUF is to create a community-endorsed single-file format fo
3030

3131
The DDUF format is also designed to be language-agnostic. While we currently provide tooling for the Python ecosystem, there's nothing stopping similar tools from being developed in JavaScript, Rust, C++, and other languages. Like GGUF or safetensors, DDUF is built to be parsable from a remote location without downloading the entire file, which will enable advanced support on the Hugging Face Hub.
3232

33-
## Key Features
33+
Its key features include the following.
3434

3535
1. **Single file** packaging.
3636
2. Based on **ZIP file format** to leverage existing tooling.
3737
3. No compression, ensuring **`mmap` compatibility** for fast loading and saving.
3838
4. **HTTP-friendly**: metadata and file structure can be fetched remotely using HTTP Range requests.
39-
5. **Flexible**: each model component is stored in its own directory, following the current `diffusers` structure.
40-
6. **Safe**: uses `safetensors` as weights saving format and prohibits nested directories to prevent ZIP-bombs.
39+
5. **Flexible**: each model component is stored in its own directory, following the current Diffusers structure.
40+
6. **Safe**: uses [Safetensors](https://huggingface.co/docs/diffusers/using-diffusers/other-formats#safetensors) as a weight-saving format and prohibits nested directories to prevent ZIP bombs.
4141

4242
## Technical specifications
4343

@@ -59,7 +59,7 @@ The `huggingface_hub` provides tooling to handle DDUF files in Python. It includ
5959

6060
### How to read a DDUF file?
6161

62-
Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path as argument. Only the metadata is read, meaning this is a lightweight call that will not make your memory explode. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally.
62+
Pass a path to `read_dduf_file` to read a DDUF file. Only the metadata is read, meaning this is a lightweight call that won't explode your memory. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally.
6363

6464
```python
6565
>>> from huggingface_hub import read_dduf_file
@@ -68,7 +68,7 @@ Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path
6868
>>> dduf_entries = read_dduf_file("FLUX.1-dev.dduf")
6969
```
7070

71-
This will return a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead.
71+
`read_dduf_file` returns a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset, and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead.
7272

7373
For instance, here is how to load the `model_index.json` content:
7474
```python
@@ -85,11 +85,12 @@ For binary files, you'll want to access the raw bytes using `as_mmap`. This retu
8585
... state_dict = safetensors.torch.load(mm) # `mm` is a bytes object
8686
```
8787

88-
Note: `as_mmap` must be used in a context manager to benefit from the memory-mapping properties.
88+
> [!TIP]
89+
> `as_mmap` must be used in a context manager to benefit from the memory-mapping properties.
8990

9091
### How to write a DDUF file?
9192

92-
A DDUF file can be exported by passing to `export_folder_as_dduf` a folder path containing a diffusion model:
93+
Pass a folder path to `export_folder_as_dduf` to export a DDUF file.
9394

9495
```python
9596
# Export a folder as a DDUF file
@@ -99,7 +100,7 @@ A DDUF file can be exported by passing to `export_folder_as_dduf` a folder path
99100

100101
This tool scans the folder, adds the relevant entries and ensures the exported file is valid. If anything goes wrong during the process, a `DDUFExportError` is raised.
101102

102-
For more flexibility, you can use [`export_entries_as_dduf`] and pass a list of files to include in the final DDUF file:
103+
For more flexibility, use [`export_entries_as_dduf`] to explicitly specify a list of files to include in the final DDUF file:
103104

104105
```python
105106
# Export specific files from the local disk.
@@ -117,7 +118,7 @@ For more flexibility, you can use [`export_entries_as_dduf`] and pass a list of
117118
... )
118119
```
119120

120-
This works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively:
121+
`export_entries_as_dduf` works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively:
121122

122123
```python
123124
(...)
@@ -146,21 +147,23 @@ ZIP provides several advantages:
146147
- Built-in file indexing
147148
- Wide language support
148149

149-
Why not use a TAR with a table of contents at the beginning of the archive? See explanation [in this comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726).
150+
### Why not use a TAR with a table of contents at the beginning of the archive?
151+
152+
See the explanation in this [comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726).
150153

151154
### Why no compression?
152155

153-
- Enables direct memory mapping of large files.
154-
- Ensures consistent and predictable remote file access.
155-
- Prevents CPU overhead during file reading.
156-
- Maintains compatibility with safetensors.
156+
- Enables direct memory mapping of large files
157+
- Ensures consistent and predictable remote file access
158+
- Prevents CPU overhead during file reading
159+
- Maintains compatibility with safetensors
157160

158161
### Can I modify a DDUF file?
159162

160163
No. For now, DDUF files are designed to be immutable. To update a model, create a new DDUF file.
161164

162165
### Which frameworks/apps support DDUFs?
163166

164-
- [diffusers](https://github.com/huggingface/diffusers)
167+
- [Diffusers](https://github.com/huggingface/diffusers)
165168

166-
We are continuously reaching out to other libraries and frameworks, if you are interested in adding support for your project, open a Discussion in the [DDUF org](https://huggingface.co/spaces/DDUF/README/discussions).
169+
We are constantly reaching out to other libraries and frameworks. If you are interested in adding support to your project, open a Discussion in the [DDUF org](https://huggingface.co/spaces/DDUF/README/discussions).

0 commit comments

Comments
 (0)