Merge branch 'main' into v0.12-release

Author: Wauplin

docs/source/how-to-cache.mdx

@@ -87,6 +87,46 @@ That `README.md` file is actually a symlink linking to the blob that has the has
 By creating the skeleton this way we open the mechanism to file sharing: if the same file was fetched in
 revision `bbbbbb`, it would have the same hash and the file would not need to be re-downloaded.
 
+### .no_exist (advanced)
+
+In addition to the `blobs`, `refs` and `snapshots` folders, you might also find a `.no_exist` folder
+in your cache. This folder keeps track of files that you've tried to download once but don't exist
+on the Hub. Its structure is the same as the `snapshots` folder with 1 subfolder per known revision:
+
+```
+<CACHE_DIR>/<REPO_NAME>/.no_exist/aaaaaa/config_that_does_not_exist.json
+```
+
+Unlike the `snapshots` folder, files are simple empty files (no symlinks). In this example,
+the file `"config_that_does_not_exist.json"` does not exist on the Hub for the revision `"aaaaaa"`.
+As it only stores empty files, this folder is neglectable is term of disk usage.
+
+So now you might wonder, why is this information even relevant?
+In some cases, a framework tries to load optional files for a model. Saving the non-existence
+of optional files makes it faster to load a model as it saves 1 HTTP call per possible optional file.
+This is for example the case in `transformers` where each tokenizer can support additional files.
+The first time you load the tokenizer on your machine, it will cache which optional files exists (and
+which doesn't) to make the loading time faster for the next initializations.
+
+To test if a file is cached locally (without making any HTTP request), you can use the [`try_to_load_from_cache`]
+helper. It will either return the filepath (if exists and cached), the object `_CACHED_NO_EXIST` (if non-existence
+is cached) or `None` (if we don't know).
+
+```python
+from huggingface_hub import try_to_load_from_cache, _CACHED_NO_EXIST
+
+filepath = try_to_load_from_cache()
+if isinstance(filepath, str):
+    # file exists and is cached
+    ...
+elif filepath is _CACHED_NO_EXIST:
+    # non-existence of file is cached
+    ...
+else:
+    # file is not cached
+    ...
+```
+
 ### In practice
 
 In practice, your cache should look like the following tree:

docs/source/package_reference/cache.mdx

@@ -6,7 +6,11 @@ for a detailed presentation of caching at HF.
 
 ## Helpers
 
-## cached_assets_path
+### try_to_load_from_cache
+
+[[autodoc]] huggingface_hub.try_to_load_from_cache
+
+### cached_assets_path
 
 [[autodoc]] huggingface_hub.cached_assets_path
 

docs/source/package_reference/hf_api.mdx

@@ -1,92 +1,79 @@
-# Hugging Face Hub API
+# HfApi Client
 
-Below is the documentation for the `HfApi` class, which serves as a Python wrapper for the Hugging Face
-Hub's API.
+Below is the documentation for the `HfApi` class, which serves as a Python wrapper for the Hugging Face Hub's API.
 
-All methods from the `HfApi` are also accessible from the package's root directly, both approaches are detailed
-below.
+All methods from the `HfApi` are also accessible from the package's root directly. Both approaches are detailed below.
 
-The following approach uses the method from the root of the package:
+Using the root method is more straightforward but the [`HfApi`] class gives you more flexibility.
+In particular, you can pass a token that will be reused in all HTTP calls. This is different
+than `huggingface-cli login` or [`login`] as the token is not persisted on the machine.
+It is also possible to provide a different endpoint or configure a custom user-agent.
 
 ```python
-from huggingface_hub import list_models
+from huggingface_hub import HfApi, list_models
 
+# Use root method
 models = list_models()
-```
-
-The following approach uses the `HfApi` class:
-
-```python
-from huggingface_hub import HfApi
-
-hf_api = HfApi()
-models = hf_api.list_models()
-```
-
-Using the [`HfApi`] class directly enables you to configure the client. In particular, a
-token can be passed to be authenticated in all API calls. This is different than
-`huggingface-cli login` or [`login`] as the token is not persisted on the machine. One
-can also specify a different endpoint than the Hugging Face's Hub (for example to interact
-with a Private Hub).
-
-```py
-from huggingface_hub import HfApi
 
+# Or configure a HfApi client
 hf_api = HfApi(
     endpoint="https://huggingface.co", # Can be a Private Hub endpoint.
     token="hf_xxx", # Token is not persisted on the machine.
 )
+models = hf_api.list_models()
 ```
 
-### HfApi
+## HfApi
 
 [[autodoc]] HfApi
 
-### RepoUrl
+## API Dataclasses
 
-[[autodoc]] huggingface_hub.hf_api.RepoUrl
-
-### ModelInfo
+### CommitInfo
 
-[[autodoc]] huggingface_hub.hf_api.ModelInfo
+[[autodoc]] huggingface_hub.hf_api.CommitInfo
 
 ### DatasetInfo
 
 [[autodoc]] huggingface_hub.hf_api.DatasetInfo
 
-### SpaceInfo
-
-[[autodoc]] huggingface_hub.hf_api.SpaceInfo
-
-### RepoFile
+### GitRefInfo
 
-[[autodoc]] huggingface_hub.hf_api.RepoFile
+[[autodoc]] huggingface_hub.hf_api.GitRefInfo
 
 ### GitRefs
 
 [[autodoc]] huggingface_hub.hf_api.GitRefs
 
-### GitRefInfo
+### ModelInfo
 
-[[autodoc]] huggingface_hub.hf_api.GitRefInfo
+[[autodoc]] huggingface_hub.hf_api.ModelInfo
 
-### CommitInfo
+### RepoFile
 
-[[autodoc]] huggingface_hub.hf_api.CommitInfo
+[[autodoc]] huggingface_hub.hf_api.RepoFile
+
+### RepoUrl
+
+[[autodoc]] huggingface_hub.hf_api.RepoUrl
+
+### SpaceInfo
+
+[[autodoc]] huggingface_hub.hf_api.SpaceInfo
 
 ### UserLikes
 
 [[autodoc]] huggingface_hub.hf_api.UserLikes
 
-## `create_commit` API
+## CommitOperation
 
 Below are the supported values for [`CommitOperation`]:
 
 [[autodoc]] CommitOperationAdd
 
 [[autodoc]] CommitOperationDelete
 
-## Hugging Face local storage
+## Token helper
 
 `huggingface_hub` stores the authentication information locally so that it may be re-used in subsequent
 methods.
@@ -95,7 +82,7 @@ It does this using the [`HfFolder`] utility, which saves data at the root of the
 
 [[autodoc]] HfFolder
 
-## Filtering helpers
+## Search helpers
 
 Some helpers to filter repositories on the Hub are available in the `huggingface_hub` package.