Fix dedup query

[seanses]

• Fix dedup query response parse logic according to the server implementation. See https://github.com/huggingface-internal/xetcas/pull/99
• Fix config setting so that global dedup is actually enabled. context
• Add clean metrics reporting for debugging (also useful for future telemetry setup). Related PR

Tested with hfxet in the below steps:

1. Upload a file, see no dedup
2. Remove local shards (rm -rf .xet)
3. Upload the same file, see 100% dedup due to global dedup

(.venv) di@di-mbp ~/tt/download % ipython
In [1]: from huggingface_hub import upload_folder, login

In [2]: login(token="...")
...

In [3]: upload_folder(folder_path="/Users/di/tt/bsf13/data/", repo_id="di/m1", repo_type="model")
...
{"timestamp":"2024-10-26T00:00:21.858541Z","level":"INFO","fields":{"message":"Cleaning file None finished in 0 s 217 ms, processed 9815362 bytes, produced 9815362 bytes after dedup."},"filename":"/Users/di/hf/xet-core/data/src/clean.rs","line_number":177}
...
Out[3]: CommitInfo(commit_url='http://localhost:5564/di/m1/commit/83f9a4fc1392e475006893cc8219270aec663820', commit_message='Upload folder using huggingface_hub', commit_description='', oid='83f9a4fc1392e475006893cc8219270aec663820', pr_url=None, repo_url=RepoUrl('http://localhost:5564/di/m1', endpoint='http://localhost:5564', repo_type='model', repo_id='di/m1'), pr_revision=None, pr_num=None)

In [4]: !rm -rf .xet

In [5]: !ls -la .xet
ls: .xet: No such file or directory

In [6]: upload_folder(folder_path="/Users/di/tt/bsf13/data/", repo_id="di/m1", repo_type="model")
...
{"timestamp":"2024-10-26T00:00:52.002688Z","level":"INFO","fields":{"message":"Cleaning file None finished in 0 s 217 ms, processed 9815362 bytes, produced 0 bytes after dedup."},"filename":"/Users/di/hf/xet-core/data/src/clean.rs","line_number":177}
...
Out[6]: CommitInfo(commit_url='http://localhost:5564/di/m1/commit/6042cd060241b5062dc86de2b259f47e9a14e7cb', commit_message='Upload folder using huggingface_hub', commit_description='', oid='6042cd060241b5062dc86de2b259f47e9a14e7cb', pr_url=None, repo_url=RepoUrl('http://localhost:5564/di/m1', endpoint='http://localhost:5564', repo_type='model', repo_id='di/m1'), pr_revision=None, pr_num=None)

[port8080]

lgtm

[assafvayner]

should this function just return 1 shard hash instead of a vec?

[seanses]

It's reserved for batch request: the function takes a list of chunk hashes and return a list of shards, though the batch part is not implemented yet.

[hoytak]

LGTM