move spec to docs (#515)

publish to hub docs out of xet-core for xet-spec. Need to merge this
first before iterating to get the github workflows working right.

Author: assafvayner

.github/workflows/build_documentation.yml

@@ -0,0 +1,20 @@
+name: Build Documentation
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+      - doc-builder*
+      - v*-release
+
+jobs:
+  build:
+    uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@main
+    with:
+      commit_sha: ${{ github.sha }}
+      package: xet-core
+      package_name: xet-spec
+      additional_args: --not_python_module
+    secrets:
+      hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }}

.github/workflows/build_pr_documentation.yml

@@ -0,0 +1,31 @@
+name: Build PR Documentation
+
+on:
+  pull_request:
+    paths:
+      - "docs/**"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  debug-workflow-name:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Echo workflow metadata
+        run: |
+          echo "github.workflow = ${{ github.workflow }}"
+          echo "github.workflow_ref = ${{ github.workflow_ref }}"
+          echo "github.workflow_sha = ${{ github.workflow_sha }}"
+          echo "github.run_id = ${{ github.run_id }}"
+          echo "github.event_name = ${{ github.event_name }}"
+          echo "github.event.number = ${{ github.event.number }}"
+  build:
+    uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@main
+    with:
+      commit_sha: ${{ github.event.pull_request.head.sha }}
+      pr_number: ${{ github.event.number }}
+      package: xet-core
+      package_name: xet-spec
+      additional_args: --not_python_module

.github/workflows/ci.yml

@@ -105,14 +105,3 @@ jobs:
         working-directory: hf_xet_wasm
         run: |
           ./build_wasm.sh
-
-  lint_markdown:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-      - name: Install markdownlint-cli
-        run: |
-          npm install -g markdownlint-cli
-      - name: Lint markdown
-        run: markdownlint spec/**/*.md --config markdownlint.toml

.github/workflows/upload_pr_documentation.yml

@@ -0,0 +1,15 @@
+name: Upload PR Documentation
+
+on:
+  workflow_run:
+    workflows: ["Build PR Documentation"]
+    types: [completed]
+
+jobs:
+  build:
+    uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@main
+    with:
+      package_name: xet-spec
+    secrets:
+      hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }}
+      comment_bot_token: ${{ secrets.COMMENT_BOT_TOKEN }}

docs/source/_toctree.yml

@@ -0,0 +1,30 @@
+- local: index
+  title: Xet Protocol Specification
+
+- title: Building a client library for xet storage
+  sections:
+    - local: upload-protocol
+      title: Upload Protocol
+    - local: download-protocol
+      title: Download Protocol
+    - local: api
+      title: CAS API
+    - local: auth
+      title: Authentication and Authorization
+    - local: file-id
+      title: Hugging Face Hub Files Conversion to Xet File ID's
+
+- title: Overall Xet architecture
+  sections:
+    - local: chunking
+      title: Content-Defined Chunking
+    - local: hashing
+      title: Hashing Methods
+    - local: file-reconstruction
+      title: File Reconstruction
+    - local: xorb
+      title: Xorb Format
+    - local: shard
+      title: Shard Format
+    - local: deduplication
+      title: Deduplication

spec/api.md renamed to docs/source/api.md

@@ -1,10 +1,10 @@
 # CAS API Documentation
 
-This document describes the HTTP API endpoints used by the CAS (Content Addressable Storage) client to interact with the remote CAS server.
+This document describes the HTTP API endpoints used by the Content Addressable Storage (CAS) client to interact with the remote CAS server.
 
 ## Authentication
 
-To authenticate, authorize, and obtain the API base URL, follow the instructions in [Authentication](../spec/auth.md).
+To authenticate, authorize, and obtain the API base URL, follow the instructions in [Authentication](./auth).
 
 ## Converting Hashes to Strings
 
@@ -18,6 +18,9 @@ For every 8 bytes in the hash (indices 0-7, 8-15, 16-23, 24-31) reverse the orde
 
 Otherwise stated, consider each 8 byte part of a hash as a little endian 64 bit unsigned integer, then concatenate the hexadecimal representation of the 4 numbers in order (each padded with 0's to 16 characters).
 
+> [!NOTE]
+> In all cases that a hash is represented as a string it is converted from a byte array to a string using this procedure.
+
 ### Example
 
 Suppose a hash value is:
@@ -38,7 +41,7 @@ It is: `07060504030201000f0e0d0c0b0a0908171615141312111f1e1d1c1b1a1918`.
 - **Method**: `GET`
 - **Parameters**:
   - `file_id`: File hash in hex format (64 lowercase hexadecimal characters).
-See [file hashes](../spec/hashing.md#file-hashes) for computing the file hash and [converting hashes to strings](../spec/api.md#converting-hashes-to-strings).
+See [file hashes](./hashing#file-hashes) for computing the file hash and [converting hashes to strings](./api#converting-hashes-to-strings).
 - **Headers**:
   - `Range`: OPTIONAL. Format: `bytes={start}-{end}` (end is inclusive).
 - **Minimum Token Scope**: `read`
@@ -53,7 +56,7 @@ See [file hashes](../spec/hashing.md#file-hashes) for computing the file hash an
   }
   ```
 
-- **Error Responses**: See [Error Cases](../spec/api.md#error-cases)
+- **Error Responses**: See [Error Cases](./api#error-cases)
   - `400 Bad Request`: Malformed `file_id` in the path. Fix the path before retrying.
   - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header.
   - `404 Not Found`: The file does not exist. Not retryable.
@@ -67,7 +70,7 @@ OPTIONAL: -H Range: "bytes=0-100000"
 
 ### Example File Reconstruction Response Body
 
-See [QueryReconstructionResponse](../spec/download_protocol.md#queryreconstructionresponse-structure) for more details in the download protocol specification.
+See [QueryReconstructionResponse](./download-protocol#queryreconstructionresponse-structure) for more details in the download protocol specification.
 
 ### 2. Query Chunk Deduplication (Global Deduplication)
 
@@ -77,11 +80,11 @@ See [QueryReconstructionResponse](../spec/download_protocol.md#queryreconstructi
 - **Parameters**:
   - `prefix`: The only acceptable prefix for the Global Deduplication API is `default-merkledb`.
   - `hash`: Chunk hash in hex format (64 lowercase hexadecimal characters).
-See [Chunk Hashes](../spec/hashing.md#chunk-hashes) to compute the chunk hash and [converting hashes to strings](../spec/api.md#converting-hashes-to-strings).
+See [Chunk Hashes](./hashing#chunk-hashes) to compute the chunk hash and [converting hashes to strings](./api#converting-hashes-to-strings).
 - **Minimum Token Scope**: `read`
 - **Body**: None.
-- **Response**: Shard format bytes (`application/octet-stream`), deserialize as a [shard](../spec/shard.md#global-deduplication).
-- **Error Responses**: See [Error Cases](../spec/api.md#error-cases)
+- **Response**: Shard format bytes (`application/octet-stream`), deserialize as a [shard](./shard#global-deduplication).
+- **Error Responses**: See [Error Cases](./api#error-cases)
   - `400 Bad Request`: Malformed hash in the path. Fix the path before retrying.
   - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header.
   - `404 Not Found`: Chunk not already tracked by global deduplication. Not retryable.
@@ -103,10 +106,10 @@ An example shard response body can be found in [Xet reference files](https://hug
 - **Parameters**:
   - `prefix`: The only acceptable prefix for the Xorb upload API is `default`.
   - `hash`: Xorb hash in hex format (64 lowercase hexadecimal characters).
-See [Xorb Hashes](../spec/hashing.md#xorb-hashes) to compute the hash, and [converting hashes to strings](../spec/api.md#converting-hashes-to-strings).
+See [Xorb Hashes](./hashing#xorb-hashes) to compute the hash, and [converting hashes to strings](./api#converting-hashes-to-strings).
 - **Minimum Token Scope**: `write`
 - **Body**: Serialized Xorb bytes (`application/octet-stream`).
-See [xorb format serialization](../spec/xorb.md).
+See [xorb format serialization](./xorb).
 - **Response**: JSON (`UploadXorbResponse`)
 
 ```json
@@ -117,7 +120,7 @@ See [xorb format serialization](../spec/xorb.md).
 
 - Note: `was_inserted` is `false` if the Xorb already exists; this is not an error.
 
-- **Error Responses**: See [Error Cases](../spec/api.md#error-cases)
+- **Error Responses**: See [Error Cases](./api#error-cases)
   - `400 Bad Request`: Malformed hash in the path, Xorb hash does not match the body, or body is incorrectly serialized.
   - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header.
   - `403 Forbidden`: Token provided but does not have a wide enough scope (for example, a `read` token was provided). Clients MUST retry with a `write` scope token.
@@ -139,7 +142,7 @@ Uploads file reconstructions and new xorb listing, serialized into the shard for
 - **Method**: `POST`
 - **Minimum Token Scope**: `write`
 - **Body**: Serialized Shard data as bytes (`application/octet-stream`).
-See [Shard format guide](../spec/shard.md#shard-upload).
+See [Shard format guide](./shard#shard-upload).
 - **Response**: JSON (`UploadShardResponse`)
 
 ```json
@@ -154,7 +157,7 @@ See [Shard format guide](../spec/shard.md#shard-upload).
 
 The value of `result` does not carry any meaning, if the upload shard API returns a `200 OK` status code, the upload was successful and the files listed are considered uploaded.
 
-- **Error Responses**: See [Error Cases](../spec/api.md#error-cases)
+- **Error Responses**: See [Error Cases](./api#error-cases)
   - `400 Bad Request`: Shard is incorrectly serialized or Shard contents failed verification.
     - Can mean that a referenced Xorb doesn't exist or the shard is too large
   - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header.

spec/auth.md renamed to docs/source/auth.md

@@ -1,6 +1,6 @@
 # Authentication and Authorization
 
-To invoke any API's mentioned in this specification a client MUST first acquire a token (and the url) to authenticate against the server which serves these API's.
+To invoke any API's mentioned in this specification a client MUST first acquire a token (and the URL) to authenticate against the server which serves these API's.
 
 The Xet protocol server uses bearer authentication via a token generated by the Hugging Face Hub (<https://huggingface.co>).
 
@@ -16,14 +16,14 @@ https://huggingface.co/api/{repo_type}s/{repo_id}/xet-{token_type}-token/{revisi
 
 **Parameters:**
 
-All parameters are required to form the url.
+All parameters are required to form the URL.
 
 - `repo_type`: Type of repository - `model`, `dataset`, or `space`
 - `repo_id`: Repository identifier in format `namespace/repo-name`
 - `token_type`: Either `read` or `write`.
 - `revision`: Git revision (branch, tag, or commit hash; default to using `main` if no specific ref is required)
 
-To understand the distinction for between `token_type` values read onwards in this document to [Token Scope](../spec/auth.md#token-scope).
+To understand the distinction for between `token_type` values read onwards in this document to [Token Scope](./auth#token-scope).
 
 **Example URLs:**
 
@@ -101,6 +101,7 @@ Here's a basic implementation flow:
 4. **Token refresh (when needed):**
    Use the same API to generate a new token.
 
+  > [!NOTE]
   > In `xet-core` we SHOULD add 30 seconds of buffer time before the provided `expiration` time to refresh the token.
 
 ## Token Scope
@@ -109,7 +110,7 @@ Xet tokens can have either a `read` or a `write` scope.
 `write` scope supersedes `read` scope and all `read` scope API's can be invoked when using a `write` scope token.
 The type of token issued is determined on the `token_type` URI path component when requesting the token from the Hugging Face Hub (see above).
 
-Revise API specification for what scope level is necessary to invoke each API (briefly, only `POST /shard` and `POST /xorb/*` API's require `write` scope).
+Check API specification for what scope level is necessary to invoke each API (briefly, only `POST /shard` and `POST /xorb/*` API's require `write` scope).
 
 The scope of the Xet tokens is limited to the repository and ref for which they were issued. To upload or download from different repositories or refs (different branches) clients MUST be issued different tokens.
 

spec/chunking.md renamed to docs/source/chunking.md

@@ -9,7 +9,7 @@ File -> | chunk 0 | chunk 1 | chunk 2 | chunk 3 | chunk 4 | chunk 5 | chunk 6 |
         +---------+---------+---------+---------+---------+---------+---------+--------------
 ```
 
-## Step-by-step algorithm (Gearhash-based CDC)
+## Step-by-step Algorithm (Gearhash-based CDC)
 
 ### Constant Parameters
 
@@ -24,15 +24,15 @@ File -> | chunk 0 | chunk 1 | chunk 2 | chunk 3 | chunk 4 | chunk 5 | chunk 6 |
 - h: 64-bit hash, initialized to 0
 - start_offset: start offset of the current chunk, initialized to 0
 
-### Per-byte update rule (Gearhash)
+### Per-byte Update Rule (Gearhash)
 
 For each input byte `b`, update the hash with 64-bit wrapping arithmetic:
 
 ```text
 h = (h << 1) + TABLE[b]
 ```
 
-### Boundary test and size constraints
+### Boundary Test and Size Constraints
 
 At each position after updating `h`, let `size = current_offset - start_offset + 1`.
 
@@ -81,39 +81,39 @@ if start_offset < len(data):
 
 ### Boundary probability and mask selection
 
-Given that MASK has 16 one-bits, for a random 64-bit hash h, the chance that all those 16 bits are zero is 1 / 2^16. On average, that means you’ll see a match about once every 64 KiB.
+Given that MASK has 16 one-bits, for a random 64-bit hash `h`, the chance that all those 16 bits are zero is 1 / 2^16. On average, that means you’ll see a match about once every 64 KiB.
 
 ### Properties
 
 - Deterministic boundaries: same content → same chunks
 - Locality: small edits only affect nearby boundaries
 - Linear time and constant memory: single 64-bit state and counters
 
-### Intuition and rationale
+### Intuition and Rationale
 
 - The table `TABLE[256]` injects pseudo-randomness per byte value so that the evolving hash `h` behaves like a random 64-bit value with respect to the mask test. This makes boundaries content-defined yet statistically evenly spaced.
 - The left shift `(h << 1)` amplifies recent bytes, helping small changes affect nearby positions without globally shifting all boundaries.
 - Resetting `h` to 0 at each boundary prevents long-range carryover and keeps boundary decisions for each chunk statistically independent.
 
-### Implementation notes
+### Implementation Notes
 
 - Only reset `h` when you emit a boundary. This ensures chunking is stable even when streaming input in pieces.
 - Apply the mask test only once `size >= MIN_CHUNK_SIZE`. This reduces the frequency of tiny chunks and stabilizes average chunk sizes.
 - MUST force a boundary at `MAX_CHUNK_SIZE` even if `(h & MASK) != 0`. This guarantees bounded chunk sizes and prevents pathological long chunks when matches are rare.
 - Use 64-bit wrapping arithmetic for `(h << 1) + TABLE[b]`. This is the behavior in the reference implementation [rust-gearhash].
 
-### Edge cases
+### Edge Cases
 
 - Tiny files: if `len(data) < MIN_CHUNK_SIZE`, the entire `data` is emitted as a single chunk.
 - Long runs without a match: if no position matches `(h & MASK) == 0` before `MAX_CHUNK_SIZE`, a boundary is forced at `MAX_CHUNK_SIZE` to cap chunk size.
 
-### Portability and determinism
+### Portability and Determinism
 
 - With a fixed `T[256]` table and mask, the algorithm is deterministic across platforms: same input → same chunk boundaries.
 - Endianness does not affect behavior because updates are byte-wise and use scalar 64-bit operations.
 - SIMD-accelerated implementations (when available) are optimizations only; they produce the same boundaries as the scalar path [rust-gearhash].
 
-## Minimum-size skip-ahead (cut-point skipping optimization)
+## Minimum-size Skip-ahead (Cut-point Skipping Optimization)
 
 Computing and testing the rolling hash at every byte is expensive for large data, and early tests inside the first few bytes of a chunk are disallowed by the `MIN_CHUNK_SIZE` constraint anyway.
 We are able to intentionally skip testing some data with cut-point skipping to accelerate scanning without affecting correctness.
@@ -141,7 +141,7 @@ The [xet-team/xet-spec-reference-files](https://huggingface.co/datasets/xet-team
 
 In the same repository in file [Electric_Vehicle_Population_Data_20250917.csv.chunks](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.chunks)
 the chunks produced out of [Electric_Vehicle_Population_Data_20250917.csv](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv) are listed.
-Each line in the file is a 64 hexadecimal hash of the chunk, followed by a space and then the number of bytes in that chunk.
+Each line in the file is a 64 hexadecimal character string version of the hash of the chunk, followed by a space and then the number of bytes in that chunk.
 
 Implementors should use the chunk lengths to determine that they are producing the right chunk boundaries for this file with their chunking implementation.
 

spec/deduplication.md renamed to docs/source/deduplication.md

@@ -23,11 +23,11 @@ A **chunk** is a variable-sized content block derived from files using Content-D
 - **Size range**: 8KB to 128KB (minimum and maximum constraints)
 - **Identification**: Each chunk is uniquely identified by its cryptographic hash (MerkleHash)
 
-[Detailed chunking description](../spec/chunking.md)
+[Detailed chunking description](./chunking)
 
-### Xorbs (Extended Object Blocks)
+### Xorbs
 
-**Xorbs** are containers that aggregate multiple chunks for efficient storage and transfer:
+**Xorbs** are objects that aggregate multiple chunks for efficient storage and transfer:
 
 - **Maximum size**: 64MB
 - **Maximum chunks**: 8,192 chunks per xorb
@@ -96,7 +96,7 @@ Xet employs a three-tiered deduplication strategy to maximize efficiency while m
 
 #### Level 3: Global Deduplication API
 
-**Scope**: Entire Xet ecosystem
+**Scope**: Entire Xet system
 **Mechanism**: Global deduplication service with HMAC protection
 **Purpose**: Discover deduplication opportunities across all users and repositories
 
@@ -143,11 +143,11 @@ They MAY know this chunk hash because they own this data, the match has made the
 ### Chunk Hash Computation
 
 Each chunk has its content hashed using a cryptographic hash function (Blake3-based MerkleHash) to create a unique identifier for content addressing.
-[See section about hashing](../spec/hashing.md#chunk-hashes).
+[See section about hashing](./hashing#chunk-hashes).
 
 ### Xorb Formation
 
-When new chunks need to be stored, they are aggregated into xorbs based on size and count limits. If adding a new chunk would exceed the maximum xorb size or chunk count, the current xorb is finalized and uploaded. [See section about xorb formation](../xorb.md)
+When new chunks need to be stored, they are aggregated into xorbs based on size and count limits. If adding a new chunk would exceed the maximum xorb size or chunk count, the current xorb is finalized and uploaded. [See section about xorb formation](./xorb)
 
 ### File Reconstruction Information
 
@@ -164,7 +164,7 @@ This information allows the system to reconstruct files by:
 2. Extracting the specific chunk ranges from each xorb
 3. Concatenating chunks in the correct order
 
-[See section about file reconstruction](../file_reconstruction.md).
+[See section about file reconstruction](./file-reconstruction).
 
 ## Fragmentation Prevention