✨ New create_commit API (#888)

* ✨ New `commit_files` API

* ✨ commit_folder utility

* 🔊 Add debug logs

* 💄 Code quality

* 🚚 Move `hf_hub_url` to `.utils.endpoint_helpers`

To avoid a circular import

* ♻ Refactoring

- Rename `commit_files` -> `create_commit`
- Refactor `upload_file`, `commit_folder` and `delete_file` to use `create_commit`
- Add `create_pr` kwarg to create a Pull Request
- Add support for File objects and `bytes` in `create_commit`
- Move `hf_hub_url` to `utils.endpoint_helpers` to avoid a circular import

* 🧪 Add tests

* 🚚 `commit_folder` -> `upload_folder`

* Re-export CommitOperations

* Re-exports

* ⏪ Revert changes to testing_constants

* 💚 Post-rebase fixes

* ♻ Refactoring and docstrings

* Refacto

* 💚 Fix imports for 3.7

* 🔥 Remove dubious type hints

* Cleaner use of SliceFileObj

* 🚚 Regroup LFS utils together

* ✅ Add tests for sha_fileobj

* 🎨 More consistent naming

* ✅ Test UploadInfo

* ✅ Add unit tests for SliceFileObj ctx manager

* 💄 Code quality 🙈

* 🩹 Fix import in commands 🤡

* 🏗 Change staging URL

* ✅ Fix tests ?

* ✅ Use `hf_hub_download` API in tests

* 📝 Add docstring for `create_commit` API

* 🚧 Multi-threaded upload in `create_commit`

* ♻ Refactoring of multi-threaded upload

Thanks @Wauplin !!

* 👌 Changes from code review

    - 📝 Add and update documentation
    - ♻ Remove TypedDict usage and introduce ad-hoc validation
    - 🧪 Re-add a removed test
    - 🎨 Simplify `CommitOperationAdd.b64content`

* ✅ Various small fixes

Fix typo

Fix URL prefix

* Add support for verify action

* ✅ Fix url prefix logic in post_lfs_batch_info

* ✨ Make SliceFileObj seekable

* 👌 Changes from code review

- 💄 HF Hub -> Hub
- 💄 Documentation conventions: single backticks
- 💄 Documentation consistency / improvements
- 🎨 `commit_summary` -> `commit_message`
Co-authored-by: Omar Sansevioro <[email protected]>

* 💄 Testing conventions

* 📝 Some more documentation

* ✨ Update update_metadata method

* ↩ Backwards-compatibility & deprecation

* 🔒 Pass `token` to Hub requests

* ✅ Test binary file upload

* ✅ Fix verify call

* ✅ Also test uploads in private repos

* ✨ Support revisions with slashes in hf_hub_url

* ✅ Add a test for creating a PR with `create_commit`

* Change deprecation version to 0.10.0

* 🎨 Code quality

* Fixup merge

* Revert "Fixup merge"

This reverts commit 3e82aad.

* ✅ Fix create_pr test

* ✅ Fix upload_folder test

* 🩹 Remove Authorization header when downloading a LFS file

* 💄 Code quality

* 📝 Deprecation period consistency

* ⚰ Remove _legacy_upload_file

* 🚚 `commit_api` -> `_commit_api`

* 🎨 API changes

- `CommitOperationAdd.upload_info` -> `CommitOperationAdd._upload_info` (make private)
- `CommitOperationAdd.fileobj` -> `CommitOperationAdd.as_file` (clearer)
- More thorough documentation for `CommitOperationAdd.validate`

* ♻ `upload_file` & `upload_folder` consistency

- Backwards-compatibility (don't change the semantic nor the return value)
- Consistent behavior between the two methods
- Add some tests

* ⏪ Revert move of  hf_hub_url method

* ⏪ re-add FileSlice and deprecate it

* 🩹 Small fixes & dedup

* ✅ Fix revision and raise RuntimeEror

* 🔗 Correct syntax for hyperlinks

* ✅ Oopsie

* 🔥 Remove FileSlice once again

* 💄 Code quality

* 📝 Add some documentation about `create_commit`

* ♻ Make validation func private

* 🩹 Fix typo

Co-authored-by: Eliott Coyac <[email protected]>

Co-authored-by: Eliott Coyac <[email protected]>

Author: 2 people

docs/source/how-to-upstream.mdx

@@ -5,6 +5,7 @@ Sharing your files and work is a very important aspect of the Hub. The `huggingf
 * Push files with a `commit` context manager.
 * Push files with the [`~Repository.push_to_hub`] function.
 * Upload very large files with [Git LFS](https://git-lfs.github.com/).
+* Push files without Git installed with [`HfApi`]
 
 Whenever you want to upload files to the Hub, you need to log in to your Hugging Face account:
 
@@ -113,4 +114,40 @@ For huge files (>5GB), you need to install a custom transfer agent for Git LFS:
 huggingface-cli lfs-enable-largefiles
 ```
 
-You should install this for each model repository that contains a model file. Once installed, you are now able to push files larger than 5GB.
+You should install this for each model repository that contains a model file. Once installed, you are now able to push files larger than 5GB.
+
+## Managing files in a repo without Git with the `create_commit` API
+
+`huggingface_hub` also offers a way to upload files to the Hub without Git installed on your system with the [`create_commit`] method of [`HfApi`].
+For example, if you want to upload two files and delete another file in a Hub repo:
+
+```py
+>>> from huggingface_hub import HfApi, CommitOperationAdd, CommitOperationDelete
+>>> api = HfApi()
+>>> operations = [
+...     CommitOperationAdd(path_in_repo="LICENSE.md", path_or_fileobj="~/repo/LICENSE.md"),
+...     CommitOperationAdd(path_in_repo="weights.h5", path_or_fileobj="~/repo/weights-final.h5"),
+...     CommitOperationDelete(path_in_repo="old-weights.h5"),
+... ]
+>>> api.create_commit(
+...     repo_id="lysandre/test-model",
+...     operations=operations,
+... )
+```
+
+[`create_commit`] uses the HTTP protocol to upload files to the Hub. It automatically takes care of uploading large files and binary files with the Git LFS protocol.
+There are currently two kind of operations supported by the [`create_commit`] method:
+
+1. [`CommitOperationAdd`] to upload a file to the Hub. If the file already exists, its content will be overwritten. It takes two arguments:
+   * `path_in_repo`: the path in the repository where the file should be uploaded
+   * `path_or_fileobj`: either a path to a file on your filesystem, or a file-like object. The content of the file to upload to the Hub.
+2. [`CommitOperationDelete`] to remove a file from a repository. It takes `path_in_repo` as an argument.
+
+Instead of [`create_commit`], you can also use the following convenience methods:
+* [`upload_file`] to upload a single file to a repo on the Hub
+* [`upload_folder`] to upload a local directory to a repo on the Hub
+* [`delete_file`] to delete a single file from a repo on the Hub 
+* [`metadata_update`] to update a repo's metadata
+
+All these methods use the `create_commit` API under the hood.
+For a more detailed description, visit the [`hf_api`] documentation page.

src/huggingface_hub/__init__.py

@@ -143,10 +143,14 @@ def __dir__():
         ],
         "file_download": ["cached_download", "hf_hub_download", "hf_hub_url"],
         "hf_api": [
+            "CommitOperation",
+            "CommitOperationAdd",
+            "CommitOperationDelete",
             "DatasetSearchArguments",
             "HfApi",
             "HfFolder",
             "ModelSearchArguments",
+            "create_commit",
             "create_repo",
             "dataset_info",
             "delete_file",
@@ -168,6 +172,7 @@ def __dir__():
             "unset_access_token",
             "update_repo_visibility",
             "upload_file",
+            "upload_folder",
             "whoami",
         ],
         "hub_mixin": ["ModelHubMixin", "PyTorchModelHubMixin"],