Skip to content

Commit 03afa2a

Browse files
authored
Non blocking git push (#315)
* Non-blocking git push Add test & typings Implement status code visualization context manager has access to commands Better warning Traditional push should also contribute to queue Add push_to-hub Docs and tests Update src/huggingface_hub/README.md Co-authored-by: Omar Sanseviero <[email protected]> Apply Omar's review * Bigger push * Simplify test * Bigger test
1 parent 4df26db commit 03afa2a

File tree

4 files changed

+344
-24
lines changed

4 files changed

+344
-24
lines changed

.gitignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,5 +127,6 @@ dmypy.json
127127
# Pyre type checker
128128
.pyre/
129129
.vscode/
130+
.idea/
130131

131-
.DS_Store
132+
.DS_Store

src/huggingface_hub/README.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,10 @@ traditional Git methods:
203203
- `git_push()`
204204
- `git_checkout(branch)`
205205

206+
The `git_push` method has a parameter `blocking` which is `True` by default. When set to `False`, the push will
207+
happen behind the scenes - which can be helpful if you would like your script to continue on while the push is
208+
happening.
209+
206210
LFS-tracking methods:
207211

208212
- `lfs_track(pattern: Union[str, List[str]], filename: bool)`. Setting
@@ -221,6 +225,7 @@ On top of these unitary methods lie some useful additional methods:
221225
tracks large files (>10Mb) with `git-lfs`. The `track_large_files` argument can
222226
be set to `False` if you wish to ignore that behavior.
223227

228+
These two methods also have support for the `blocking` parameter.
224229

225230
Examples using the `commit` context manager:
226231
```python
@@ -236,6 +241,46 @@ Examples using the `commit` context manager:
236241
... torch.save(model.state_dict(), "model.pt")
237242
```
238243

244+
### Non-blocking behavior
245+
246+
The pushing methods have access to a `blocking` boolean parameter to indicate whether the push should happen
247+
asynchronously.
248+
249+
In order to see if the push has finished or its status code (to spot a failure), one should use the `command_queue`
250+
property on the `Repository` object.
251+
252+
For example:
253+
254+
```python
255+
from huggingface_hub import Repository
256+
257+
repo = Repository("<local_folder>", clone_from="<user>/<model_name>")
258+
259+
with repo.commit("Commit message", blocking=False):
260+
# Save data
261+
262+
last_command = repo.command_queue[-1]
263+
264+
# Status of the push command
265+
last_command.status
266+
# Will return the status code
267+
# -> -1 will indicate the push is still ongoing
268+
# -> 0 will indicate the push has completed successfully
269+
# -> non-zero code indicates the error code if there was an error
270+
271+
# if there was an error, the stderr may be inspected
272+
last_command.stderr
273+
274+
# Whether the command finished or if it is still ongoing
275+
last_command.is_done
276+
277+
# Whether the command errored-out.
278+
last_command.failed
279+
```
280+
281+
When using `blocking=False`, the commands will be tracked and your script will exit only when all pushes are done, even
282+
if other errors happen in your script (a failed push counts as done).
283+
239284

240285
### Need to upload very large (>5GB) files?
241286

0 commit comments

Comments
 (0)