Skip to content

Commit c04bc22

Browse files
committed
Moving toctree around
1 parent e8c859c commit c04bc22

File tree

2 files changed

+39
-7
lines changed

2 files changed

+39
-7
lines changed

docs/hub/_toctree.yml

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,39 @@
4848
- local: storage-limits
4949
title: Storage Limits
5050
- local: storage-backends
51-
title: Storage Backends
51+
title: Storage Backend - Xet
52+
sections:
53+
- local: xet/index
54+
title: Open Source Xet Protocol
55+
sections:
56+
- local: xet/index
57+
title: Building a client library for xet storage
58+
sections:
59+
- local: xet/upload-protocol
60+
title: Upload Protocol
61+
- local: xet/download-protocol
62+
title: Download Protocol
63+
- local: xet/api
64+
title: CAS API
65+
- local: xet/auth
66+
title: Authentication and Authorization
67+
- local: xet/file-id
68+
title: Hugging Face Hub Files Conversion to Xet File ID's
69+
- local: xet/index
70+
title: Overall Xet architecture
71+
sections:
72+
- local: xet/chunking
73+
title: Content-Defined Chunking
74+
- local: xet/hashing
75+
title: Hashing Methods
76+
- local: xet/file-reconstruction
77+
title: File Reconstruction
78+
- local: xet/xorb
79+
title: Xorb Format
80+
- local: xet/shard
81+
title: Shard Format
82+
- local: xet/deduplication
83+
title: Deduplication
5284
- local: repositories-pull-requests-discussions
5385
title: Pull Requests & Discussions
5486
- local: notifications

docs/hub/storage-backends.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,12 @@ The Hub's [current recommendation](https://huggingface.co/docs/hub/storage-limit
9999

100100
For more details, refer to the [From Files to Chunks](https://huggingface.co/blog/from-files-to-chunks) and [From Chunks to Blocks](https://huggingface.co/blog/from-chunks-to-blocks) blog posts, or the [Git is for Data](https://www.cidrdb.org/cidr2023/papers/p43-low.pdf) paper by Low et al. that served as the launch point for XetHub prior to being acquired by Hugging Face.
101101

102+
### Open Source Xet Protocol
103+
104+
The Xet storage backend is built on an open source protocol that enables efficient, chunk-based storage and retrieval of large files. This protocol provides the foundation for the deduplication and performance benefits described throughout this documentation.
105+
106+
For detailed technical specifications about the Xet protocol, including API endpoints, authentication mechanisms, chunking algorithms, and file reconstruction processes, see the [Xet Protocol Specification](./xet/index).
107+
102108
### Backward Compatibility with LFS
103109

104110
Xet storage provides a seamless transition for existing Hub repositories. It isn't necessary to know if the Xet backend is involved at all. Xet-backed repositories continue to use the Git LFS pointer file format; the addition of the `Xet backed hash` is only added to the web interface as a convenience. Practically, this means existing repos and newly created repos will not look any different if you do a `bare clone` of them. Each of the large files (or binary files) will continue to have a pointer file that matches the Git LFS pointer file specification.
@@ -120,9 +126,3 @@ The legacy storage system on the Hub, Git LFS utilizes many of the same conventi
120126
The primary limitation of Git LFS is its file-centric approach to deduplication. Any change to a file, irrespective of how large of small that change is, means the entire file is versioned - incurring significant overheads in file transfers as the entire file is uploaded (if committing to a repository) or downloaded (if pulling the latest version to your machine).
121127

122128
This leads to a worse developer experience along with a proliferation of additional storage.
123-
124-
## Open Source Xet Protocol
125-
126-
The Xet storage backend is built on an open source protocol that enables efficient, chunk-based storage and retrieval of large files. This protocol provides the foundation for the deduplication and performance benefits described throughout this documentation.
127-
128-
For detailed technical specifications about the Xet protocol, including API endpoints, authentication mechanisms, chunking algorithms, and file reconstruction processes, see the [Xet Protocol Specification](./xet/index).

0 commit comments

Comments
 (0)