[dev 🤫] Basic bucket API - not meant to be merged

[Wauplin]

Currently only for debug/tests purposes. Mimics server API. In practice we would refactor some logic, move parts to a _bucket.py module, create a high-level interface, build a CLI, etc. Not in the scope for this PR.

Related to (private) PR in moon-landing https://github.com/huggingface-internal/moon-landing/pull/16332

Disclaimer: CLI implementation is AI-generated and not reviewed. Nothing is set in stone yet.

---

How to use

1. Run moon-landing locally
2. create token from julien-c dev account
3. edit token in bucket.py and run python bucket.py

Result

# Create bucket
{'url': 'http://localhost:5564/buckets/julien-c/test-bucket', 'id': 'julien-c/test-bucket'}

# Get bucket info
{'_id': '69665cc73378355fe6b7a39c', 'id': 'julien-c/test-bucket', 'private': False, 'createdAt': '2026-01-13T14:55:03.290Z', 'lastModified': '2026-01-13T14:55:03.290Z', 'usedStorage': 0}

# List buckets (non-empty)
[{'_id': '69665cc73378355fe6b7a39c', 'id': 'julien-c/test-bucket', 'private': False, 'createdAt': '2026-01-13T14:55:03.290Z', 'lastModified': '2026-01-13T14:55:03.290Z'}]

# Delete bucket
None

# List buckets (empty)
[]

# Create private bucket
{'url': 'http://localhost:5564/buckets/julien-c/test-bucket-private', 'id': 'julien-c/test-bucket-private'}

# Get private bucket info (with token)
{'_id': '69665cc73378355fe6b7a39d', 'id': 'julien-c/test-bucket-private', 'private': True, 'createdAt': '2026-01-13T14:55:03.920Z', 'lastModified': '2026-01-13T14:55:03.920Z', 'usedStorage': 0}

# Get private bucket info (without token)
Client error '401 Unauthorized' for url 'http://localhost:5173/api/buckets/julien-c/test-bucket-private' (Request ID: f61f2de0-735e-4fae-ba2a-0a7bd462ddbc)
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

Invalid username or password.

# Create bucket again
{'url': 'http://localhost:5564/buckets/julien-c/test-bucket-with-files', 'id': 'julien-c/test-bucket-with-files'}

# List bucket tree (empty)
[]

# Upload file to bucket
Processing Files (10 / 10)    : |                                                                             | 1.00MB /  0.00B,  0.00B/s  
New Data Upload               : |                                                                             | 1.00MB /  0.00B,  0.00B/s  
  ...mp191oa91v/xL9QbouXlF.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/2RIsp7HvSQ.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/wfRWTruuPl.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/7buXEkcqJ8.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/18wKwVl5ej.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/BYk7VfSsms.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/00KJOmIdn0.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/IHWgr8rRBc.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/cgx9drDoWZ.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
  ...mp191oa91v/JVjhcL8dbW.bin: 100%|█████████████████████████████████████████████████████████████████████████|  100kB /  100kB            
{'success': True}

# List bucket tree (with files)
Found 10 objects in bucket: ['00KJOmIdn0.bin', '18wKwVl5ej.bin', '2RIsp7HvSQ.bin', '7buXEkcqJ8.bin', 'BYk7VfSsms.bin', 'IHWgr8rRBc.bin', 'JVjhcL8dbW.bin', 'cgx9drDoWZ.bin', 'wfRWTruuPl.bin', 'xL9QbouXlF.bin']

# Head bucket /resolve 00KJOmIdn0.bin
{'vary': 'Origin', 'x-powered-by': 'huggingface-moon', 'x-request-id': '215ed9c1-5c45-4b3b-b77c-38fee5c4c278', 'ratelimit': '"pages";r=999;t=232', 'ratelimit-policy': '"fixed window";"pages";q=1000;w=300', 'cross-origin-opener-policy': 'same-origin', 'referrer-policy': 'strict-origin-when-cross-origin', 'content-length': '100021', 'etag': '"5aca5ec2c9e73623b697dc4e12b7cc876da5b3dc200c1c9ebc48a507f83b8649"', 'x-xet-hash': '5aca5ec2c9e73623b697dc4e12b7cc876da5b3dc200c1c9ebc48a507f83b8649', 'last-modified': 'Tue, 13 Jan 2026 14:55:04 GMT', 'content-type': 'application/octet-stream', 'link': '<http://localhost:5564/api/buckets/julien-c/test-bucket-with-files/xet-read-token/latest>; rel="xet-auth", <http://localhost:4884/v1/reconstructions/5aca5ec2c9e73623b697dc4e12b7cc876da5b3dc200c1c9ebc48a507f83b8649>; rel="xet-reconstruction-info"', 'date': 'Tue, 13 Jan 2026 14:55:05 GMT', 'connection': 'close'}

# Delete first 3 files
{'success': True}

# Head bucket /resolve 00KJOmIdn0.bin (doesn't exist anymore)
Client error '404 Not Found' for url 'http://localhost:5173/buckets/julien-c/test-bucket-with-files/resolve/latest/00KJOmIdn0.bin' (Request ID: b2a1e4e2-17b6-4533-8634-a1f9b86ec96f)
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

File not found

# List bucket tree (with files)
Found 7 objects in bucket: ['7buXEkcqJ8.bin', 'BYk7VfSsms.bin', 'IHWgr8rRBc.bin', 'JVjhcL8dbW.bin', 'cgx9drDoWZ.bin', 'wfRWTruuPl.bin', 'xL9QbouXlF.bin']

# Download file 7buXEkcqJ8.bin from bucket
Downloading bucket file: 100%|███████████████████████████████████████████████████████████████████████████| 100k/100k [00:00<00:00, 894kB/s]
File downloaded to 7buXEkcqJ8.bin
Local file size: 100021 bytes

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Wauplin]

Demo of a first version of the CLI:

bucket_demo.mp4

[Wauplin]

Current CLI usage.

hf sync is an alias for hf bucket sync

Usage:
    hf bucket --help

    # Create a bucket
    hf bucket create user/my-bucket
    hf bucket create user/my-bucket --private
    hf bucket create user/my-bucket --exist-ok

    # Delete a bucket
    hf bucket delete user/my-bucket
    hf bucket delete user/my-bucket --yes
    hf bucket delete user/my-bucket --missing-ok

    # List files in a bucket
    hf bucket ls hf://buckets/user/my-bucket
    hf bucket ls hf://buckets/user/my-bucket/models

    # List files with human-readable sizes
    hf bucket ls hf://buckets/user/my-bucket -h

    # List files in tree format
    hf bucket ls hf://buckets/user/my-bucket --tree

    # Sync files between local and bucket
    hf bucket sync ./data hf://buckets/user/my-bucket
    hf bucket sync hf://buckets/user/my-bucket ./data

    # Delete destination files not in source
    hf bucket sync ./data hf://buckets/user/my-bucket --delete

    # Ignore modification times, compare only sizes
    hf bucket sync ./data hf://buckets/user/my-bucket --ignore-times

    # Ignore sizes, compare only modification times
    hf bucket sync ./data hf://buckets/user/my-bucket --ignore-sizes

    # With filters
    hf bucket sync hf://buckets/user/my-bucket ./data --include "*.safetensors" --exclude "*.tmp"
    hf bucket sync ./data hf://buckets/user/my-bucket --filter-from filters.txt

    # Only update existing files (skip new files)
    hf bucket sync ./data hf://buckets/user/my-bucket --existing

    # Only create new files (skip existing files)
    hf bucket sync ./data hf://buckets/user/my-bucket --ignore-existing

    # Safe review workflow
    hf bucket sync ./data hf://buckets/user/my-bucket --plan sync-plan.jsonl
    hf bucket sync --apply sync-plan.jsonl

    # Verbose output with detailed logging
    hf bucket sync ./data hf://buckets/user/my-bucket --verbose

    # Quiet mode with minimal output
    hf bucket sync ./data hf://buckets/user/my-bucket --quiet