Allowlist and denylist when uploading a folder (#994)

* Allowlist and denylist when uploading a folder

* renaming in documentation

* wrong arg name

* add test case

* documentation

* do not re-run tests when updating doc

* doc

* Deprecate allow_regex and ignore_regex usage

* FIX merge problem

* move paths utils to private module + indentation in doc

Author: Wauplin

.github/workflows/python-tests.yml

@@ -5,8 +5,12 @@ on:
     branches:
       - main
       - ci_*
+    paths-ignore:
+      - "docs/**"
   pull_request:
     types: [assigned, opened, synchronize, reopened]
+    paths-ignore:
+      - "docs/**"
 
 env:
   HUGGINGFACE_CO_STAGING: yes

docs/source/how-to-downstream.mdx

@@ -85,28 +85,28 @@ already know the file names you need.
 
 However, you don't always want to download the contents of an entire repository with
 [`snapshot_download`]. Even if you don't know the file name, you can download specific
-files if you know the file type with `allow_regex` and `ignore_regex`. Use the
-`allow_regex` and `ignore_regex` arguments to specify which files to download. These
-parameters accept either a single regex or a list of regexes. 
+files if you know the file type with `allow_patterns` and `ignore_patterns`. Use the
+`allow_patterns` and `ignore_patterns` arguments to specify which files to download. These
+parameters accept either a single pattern or a list of patterns.
 
-The regex matching is based on
-[`fnmatch`](https://docs.python.org/3/library/fnmatch.html), which provides support for
-Unix shell-style wildcards.
+Patterns are Standard Wildcards (globbing patterns) as documented
+[here](https://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm). The pattern
+matching is based on [`fnmatch`](https://docs.python.org/3/library/fnmatch.html).
 
-For example, you can use `allow_regex` to only download JSON configuration files:
+For example, you can use `allow_patterns` to only download JSON configuration files:
 
 ```python
 >>> from huggingface_hub import snapshot_download
->>> snapshot_download(repo_id="lysandre/arxiv-nlp", allow_regex="*.json")
+>>> snapshot_download(repo_id="lysandre/arxiv-nlp", allow_patterns="*.json")
 ```
 
-On the other hand, `ignore_regex` can exclude certain files from being downloaded. The
+On the other hand, `ignore_patterns` can exclude certain files from being downloaded. The
 following example ignores the `.msgpack` and `.h5` file extensions:
 
 ```python
 >>> from huggingface_hub import snapshot_download
->>> snapshot_download(repo_id="lysandre/arxiv-nlp", ignore_regex=["*.msgpack", "*.h5"])
+>>> snapshot_download(repo_id="lysandre/arxiv-nlp", ignore_patterns=["*.msgpack", "*.h5"])
 ```
 
-Passing a regex can be especially useful when repositories contain files that are never
+Passing a pattern can be especially useful when repositories contain files that are never
 expected to be downloaded by [`snapshot_download`].

docs/source/how-to-upstream.mdx

@@ -39,24 +39,31 @@ Specify the path of the file to upload, where you want to upload the file to in
 ```py
 >>> from huggingface_hub import HfApi
 >>> api = HfApi()
->>> api.upload_file(path_or_fileobj="/path/to/local/folder/README.md", 
-...                 path_in_repo="README.md", 
-...                 repo_id="username/test-dataset",
-...                 repo_type="dataset",
+>>> api.upload_file(
+...     path_or_fileobj="/path/to/local/folder/README.md",
+...     path_in_repo="README.md",
+...     repo_id="username/test-dataset",
+...     repo_type="dataset",
 ... )
 ```
 
 ### Upload a folder
 
 Use the [`upload_folder`] function to upload a local folder to an existing repository. Specify the path of the local folder to upload, where you want to upload the folder to in the repository, and the name of the repository you want to add the folder to. Depending on your repository type, you can optionally set the repository type as a `dataset`, `model`, or `space`.
 
+Use the `allow_patterns` and `ignore_patterns` arguments to specify which files to upload. These parameters accept either a single pattern or a list of patterns.
+Patterns are Standard Wildcards (globbing patterns) as documented [here](https://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm).
+If both `allow_patterns` and `ignore_patterns` are provided, both constraints apply. By default, all files from the folder are uploaded.
+
 ```py
 >>> from huggingface_hub import HfApi
 >>> api = HfApi()
->>> api.upload_folder(folder_path="/path/to/local/folder",
-...                   path_in_repo="my-dataset/train",
-...                   repo_id="username/test-dataset",
-...                   repo_type="dataset",
+>>> api.upload_folder(
+...     folder_path="/path/to/local/folder",
+...     path_in_repo="my-dataset/train",
+...     repo_id="username/test-dataset",
+...     repo_type="dataset",
+...     ignore_patterns="**/logs/*.txt",
 ... )
 ```
 

src/huggingface_hub/_snapshot_download.py

@@ -1,43 +1,22 @@
 import os
-from fnmatch import fnmatch
 from pathlib import Path
 from typing import Dict, List, Optional, Union
 
 from .constants import DEFAULT_REVISION, HUGGINGFACE_HUB_CACHE, REPO_TYPES
 from .file_download import REGEX_COMMIT_HASH, hf_hub_download, repo_folder_name
 from .hf_api import HfApi, HfFolder
-from .utils import logging
+from .utils import filter_repo_objects, logging
+from .utils._deprecation import _deprecate_arguments
 
 
 logger = logging.get_logger(__name__)
 
 
-def _filter_repo_files(
-    *,
-    repo_files: List[str],
-    allow_regex: Optional[Union[List[str], str]] = None,
-    ignore_regex: Optional[Union[List[str], str]] = None,
-) -> List[str]:
-    allow_regex = [allow_regex] if isinstance(allow_regex, str) else allow_regex
-    ignore_regex = [ignore_regex] if isinstance(ignore_regex, str) else ignore_regex
-    filtered_files = []
-    for repo_file in repo_files:
-        # if there's an allowlist, skip download if file does not match any regex
-        if allow_regex is not None and not any(
-            fnmatch(repo_file, r) for r in allow_regex
-        ):
-            continue
-
-        # if there's a denylist, skip download if file does matches any regex
-        if ignore_regex is not None and any(
-            fnmatch(repo_file, r) for r in ignore_regex
-        ):
-            continue
-
-        filtered_files.append(repo_file)
-    return filtered_files
-
-
+@_deprecate_arguments(
+    version="0.12",
+    deprecated_args={"allow_regex", "ignore_regex"},
+    custom_message="Please use `allow_patterns` and `ignore_patterns` instead.",
+)
 def snapshot_download(
     repo_id: str,
     *,
@@ -54,6 +33,8 @@ def snapshot_download(
     local_files_only: Optional[bool] = False,
     allow_regex: Optional[Union[List[str], str]] = None,
     ignore_regex: Optional[Union[List[str], str]] = None,
+    allow_patterns: Optional[Union[List[str], str]] = None,
+    ignore_patterns: Optional[Union[List[str], str]] = None,
 ) -> str:
     """Download all files of a repo.
 
@@ -98,10 +79,10 @@ def snapshot_download(
         local_files_only (`bool`, *optional*, defaults to `False`):
             If `True`, avoid downloading the file and return the path to the
             local cached file if it exists.
-        allow_regex (`list of str`, `str`, *optional*):
-            If provided, only files matching this regex are downloaded.
-        ignore_regex (`list of str`, `str`, *optional*):
-            If provided, files matching this regex are not downloaded.
+        allow_patterns (`List[str]` or `str`, *optional*):
+            If provided, only files matching at least one pattern are downloaded.
+        ignore_patterns (`List[str]` or `str`, *optional*):
+            If provided, files matching any of the patterns are not downloaded.
 
     Returns:
         Local folder path (string) of repo snapshot
@@ -119,7 +100,6 @@ def snapshot_download(
 
     </Tip>
     """
-
     if cache_dir is None:
         cache_dir = HUGGINGFACE_HUB_CACHE
     if revision is None:
@@ -151,6 +131,13 @@ def snapshot_download(
         cache_dir, repo_folder_name(repo_id=repo_id, repo_type=repo_type)
     )
 
+    # TODO: remove these 4 lines in version 0.12
+    #       Deprecated code to ensure backward compatibility.
+    if allow_regex is not None:
+        allow_patterns = allow_regex
+    if ignore_regex is not None:
+        ignore_patterns = ignore_regex
+
     # if we have no internet connection we will look for an
     # appropriate folder in the cache
     # If the specified revision is a commit hash, look inside "snapshots".
@@ -181,10 +168,10 @@ def snapshot_download(
     repo_info = _api.repo_info(
         repo_id=repo_id, repo_type=repo_type, revision=revision, token=token
     )
-    filtered_repo_files = _filter_repo_files(
-        repo_files=[f.rfilename for f in repo_info.siblings],
-        allow_regex=allow_regex,
-        ignore_regex=ignore_regex,
+    filtered_repo_files = filter_repo_objects(
+        items=[f.rfilename for f in repo_info.siblings],
+        allow_patterns=allow_patterns,
+        ignore_patterns=ignore_patterns,
     )
     commit_hash = repo_info.sha
     snapshot_folder = os.path.join(storage_folder, "snapshots", commit_hash)

src/huggingface_hub/hf_api.py

@@ -51,7 +51,7 @@
     REPO_TYPES_URL_PREFIXES,
     SPACES_SDK_TYPES,
 )
-from .utils import logging
+from .utils import filter_repo_objects, logging
 from .utils._deprecation import _deprecate_positional_args
 from .utils._errors import (
     _raise_convert_bad_request,
@@ -2215,6 +2215,8 @@ def upload_folder(
         revision: Optional[str] = None,
         create_pr: Optional[bool] = None,
         parent_commit: Optional[str] = None,
+        allow_patterns: Optional[Union[List[str], str]] = None,
+        ignore_patterns: Optional[Union[List[str], str]] = None,
     ):
         """
         Upload a local folder to the given repo. The upload is done
@@ -2224,6 +2226,13 @@ def upload_folder(
         The structure of the folder will be preserved. Files with the same name
         already present in the repository will be overwritten, others will be left untouched.
 
+        Use the `allow_patterns` and `ignore_patterns` arguments to specify which files
+        to upload. These parameters accept either a single pattern or a list of
+        patterns. Patterns are Standard Wildcards (globbing patterns) as documented
+        [here](https://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm). If both
+        `allow_patterns` and `ignore_patterns` are provided, both constraints apply. By
+        default, all files from the folder are uploaded.
+
         Uses `HfApi.create_commit` under the hood.
 
         Args:
@@ -2259,7 +2268,10 @@ def upload_folder(
                 If specified and `create_pr` is `True`, the pull request will be created from `parent_commit`.
                 Specifying `parent_commit` ensures the repo has not changed before committing the changes, and can be
                 especially useful if the repo is updated / committed to concurrently.
-
+            allow_patterns (`List[str]` or `str`, *optional*):
+                If provided, only files matching at least one pattern are uploaded.
+            ignore_patterns (`List[str]` or `str`, *optional*):
+                If provided, files matching any of the patterns are not uploaded.
 
         Returns:
             `str`: A URL to visualize the uploaded folder on the hub
@@ -2284,6 +2296,7 @@ def upload_folder(
         ...     repo_id="username/my-dataset",
         ...     repo_type="datasets",
         ...     token="my_token",
+        ...     ignore_patterns="**/logs/*.txt",
         ... )
         # "https://huggingface.co/datasets/username/my-dataset/tree/main/remote/experiment/checkpoints"
 
@@ -2312,7 +2325,12 @@ def upload_folder(
             else f"Upload {path_in_repo} with huggingface_hub"
         )
 
-        files_to_add = _prepare_upload_folder_commit(folder_path, path_in_repo)
+        files_to_add = _prepare_upload_folder_commit(
+            folder_path,
+            path_in_repo,
+            allow_patterns=allow_patterns,
+            ignore_patterns=ignore_patterns,
+        )
 
         pr_url = self.create_commit(
             repo_type=repo_type,
@@ -3232,9 +3250,16 @@ def delete_token(cls):
 
 
 def _prepare_upload_folder_commit(
-    folder_path: str, path_in_repo: str
+    folder_path: str,
+    path_in_repo: str,
+    allow_patterns: Optional[Union[List[str], str]] = None,
+    ignore_patterns: Optional[Union[List[str], str]] = None,
 ) -> List[CommitOperationAdd]:
-    """Generate the list of Add operations for a commit to upload a folder."""
+    """Generate the list of Add operations for a commit to upload a folder.
+
+    Files not matching the `allow_patterns` (allowlist) and `ignore_patterns` (denylist)
+    constraints are discarded.
+    """
     folder_path = os.path.normpath(os.path.expanduser(folder_path))
     if not os.path.isdir(folder_path):
         raise ValueError(f"Provided path: '{folder_path}' is not a directory")
@@ -3252,7 +3277,15 @@ def _prepare_upload_folder_commit(
                     ).replace(os.sep, "/"),
                 )
             )
-    return files_to_add
+
+    return list(
+        filter_repo_objects(
+            files_to_add,
+            allow_patterns=allow_patterns,
+            ignore_patterns=ignore_patterns,
+            key=lambda x: x.path_in_repo,
+        )
+    )
 
 
 def _parse_revision_from_pr_url(pr_url: str) -> str:

src/huggingface_hub/hub_mixin.py

@@ -2,7 +2,7 @@
 import os
 import tempfile
 from pathlib import Path
-from typing import Dict, Optional, Union
+from typing import Dict, List, Optional, Union
 
 import requests
 from huggingface_hub import hf_api
@@ -268,6 +268,8 @@ def push_to_hub(
         token: Optional[str] = None,
         branch: Optional[str] = None,
         create_pr: Optional[bool] = None,
+        allow_patterns: Optional[Union[List[str], str]] = None,
+        ignore_patterns: Optional[Union[List[str], str]] = None,
         # TODO (release 0.12): signature must be the following
         # repo_id: str,
         # *,
@@ -278,10 +280,15 @@ def push_to_hub(
         # branch: Optional[str] = None,
         # create_pr: Optional[bool] = None,
         # config: Optional[dict] = None,
+        # allow_patterns: Optional[Union[List[str], str]] = None,
+        # ignore_patterns: Optional[Union[List[str], str]] = None,
     ) -> str:
         """
         Upload model checkpoint to the Hub.
 
+        Use `allow_patterns` and `ignore_patterns` to precisely filter which files
+        should be pushed to the hub. See [`upload_folder`] reference for more details.
+
         Parameters:
             repo_id (`str`, *optional*):
                 Repository name to which push.
@@ -304,6 +311,10 @@ def push_to_hub(
                 Defaults to `False`.
             config (`dict`, *optional*):
                 Configuration object to be saved alongside the model weights.
+            allow_patterns (`List[str]` or `str`, *optional*):
+                If provided, only files matching at least one pattern are pushed.
+            ignore_patterns (`List[str]` or `str`, *optional*):
+                If provided, files matching any of the patterns are not pushed.
 
         Returns:
             The url of the commit of your model in the given repository.
@@ -334,6 +345,8 @@ def push_to_hub(
                     commit_message=commit_message,
                     revision=branch,
                     create_pr=create_pr,
+                    allow_patterns=allow_patterns,
+                    ignore_patterns=ignore_patterns,
                 )
 
         # If the repo id is None, it means we use the deprecated version using Git