[Jobs] Add `huggingface-cli jobs` commands #3211

lhoestq · 2025-07-10T15:17:34Z

HuggingFaceDocBuilderDev · 2025-07-10T15:45:07Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

src/huggingface_hub/commands/jobs/_cli_utils.py

lhoestq · 2025-07-11T14:11:53Z

This is ready for review for the launch in the coming days ! Would be cool to do a release right after we merge

Btw I integrated your addition @davanstrien from lhoestq/hfjobs#8 and added some useful uv options: --with and --python (we could add more later if needed)

setup.py

src/huggingface_hub/commands/jobs/uv.py

julien-c · 2025-07-12T17:05:52Z

src/huggingface_hub/commands/jobs/uv.py

It's fine as an experiment, but not a huge fan of the local file uploading to a remote repo..

Is there any way to either:

pass the file content as an argument (string) to uv (and thus to the Jobs creation API)

ask the infra team to add a new feature to the Jobs creation API where you can a dict of file name to file contents and they are exposed to the docker command? (not sure if it's feasible @christophe-rannou)

pass the file content as an argument (string) to uv (and thus to the Jobs creation API)

I don't think this is directly possible in UV at the moment.

ask the infra team to add a new feature to the Jobs creation API where you can a dict of file name to file contents and they are exposed to the docker command? (not sure if it's feasible @christophe-rannou)

Think this would be nice if it was possible. @christophe-rannou, would this be difficult to implement?

Part of the logic of doing using a repo as a backend was to open up options to explore approaches where you could do something like

huggingface-cli jobs uv run --from-repo davanstrien/nice-data-generation-pipeline

I think for that to fully make sense, it would probably also be better to have a "generic" or "code" repo type rather than using a dataset as the storage repo.

Co-authored-by: Julien Chaumond <[email protected]>

Wauplin · 2025-07-15T16:10:49Z

As I high-level comment, it'd be good to have all the API logic added to HfApi (and therefore callable from a Python script) and the CLI logic (e.g. args, result formatting, etc.) kept in the current ./src/huggingface_hub/commands/jobs folder. @lhoestq Let us know if you have bandwidth to work on this or if you want some help

lhoestq · 2025-07-15T16:41:31Z

As I high-level comment, it'd be good to have all the API logic added to HfApi (and therefore callable from a Python script) and the CLI logic (e.g. args, result formatting, etc.) kept in the current ./src/huggingface_hub/commands/jobs folder. @lhoestq Let us know if you have bandwidth to work on this or if you want some help

I can take care of this for tomorrow

hanouticelina

Very nice UX, I like it! 🔥 i left some initial comments.
As mentioned by @Wauplin, let's centralize the logic into HfApi:

HfApi.run_job(...)
HfApi.list_jobs(...)
HfApi.inspect_job(...)
HfApi.cancel_job(...)
HfApi.fetch_job_logs(...)

that way, the CLI subcommands become lightweight wrappers and maybe we can put all the sub parsers (run, ps, inspect, logs, cancel, uv) inside one src/huggingface_hub/commands/jobs.py file.

src/huggingface_hub/commands/jobs/cancel.py

src/huggingface_hub/commands/jobs/run.py

src/huggingface_hub/commands/jobs/inspect.py

src/huggingface_hub/commands/jobs/uv.py

src/huggingface_hub/commands/jobs/logs.py

src/huggingface_hub/commands/jobs/cancel.py

lhoestq · 2025-07-22T16:25:24Z

I took your comments into account and added namespace in the HfApimethods and in the CLI :)

I also added the documentation page

davanstrien · 2025-07-23T07:42:25Z

Imo it would be good to still have support for passing an image when using uv run. Many ai focused images have uv included now and it can make the setup much more reliable i.e. for vllm it works out of the box using their default image. If we pass the astral image there is still a lot more steps to get things working smoothly

lhoestq · 2025-07-23T12:37:56Z

I added the image arg to run_uv_job(). And also --image to huggingface-cli jobs uv run :)

Wauplin · 2025-07-23T12:44:30Z

I also just pushed a commit to use SpaceHardware as a list of possible flavors (I forgot we had this one^^) eaaa6a1. Hopefully makes it easier for users to know what they can chose from:

✗ huggingface-cli jobs run --help                                                     
usage: huggingface-cli <command> [<args>] jobs run [-h] [-e ENV] [-s SECRETS] [--env-file ENV_FILE] [--secrets-file SECRETS_FILE] [--flavor FLAVOR] [--timeout TIMEOUT] [-d]
                                                   [--namespace NAMESPACE] [--token TOKEN]
                                                   image ...

positional arguments:
  image                 The Docker image to use.
  command               The command to run.

options:
  -h, --help            show this help message and exit
  -e ENV, --env ENV     Set environment variables.
  -s SECRETS, --secrets SECRETS
                        Set secret environment variables.
  --env-file ENV_FILE   Read in a file of environment variables.
  --secrets-file SECRETS_FILE
                        Read in a file of secret environment variables.
  --flavor FLAVOR       Flavor for the hardware, as in HF Spaces. Defaults to `cpu-basic`. Possible values: cpu-basic, cpu-upgrade, cpu-xl, zero-a10g, t4-small, t4-medium, l4x1, l4x4,
                        l40sx1, l40sx4, l40sx8, a10g-small, a10g-large, a10g-largex2, a10g-largex4, a100-large, h100, h100x8.
  --timeout TIMEOUT     Max duration: int/float with s (seconds, default), m (minutes), h (hours) or d (days).
  -d, --detach          Run the Job in the background and print the Job ID.
  --namespace NAMESPACE
                        The namespace where the Job will be created. Defaults to the current user's namespace.
  --token TOKEN         A User Access Token generated from https://huggingface.co/settings/tokens

Wauplin

Finally approved ✔️ 😄
Thanks a lot @lhoestq @davanstrien for the work on this tool! 🔥

julien-c · 2025-07-23T12:51:08Z

exclude zero-a10g from the flavors possibly because i don't think this is a valid hardware here

Wauplin · 2025-07-23T13:00:29Z

exclude zero-a10g from the flavors possibly because i don't think this is a valid hardware here

Good call, removed in e6043ae

Wauplin

👍

lhoestq · 2025-07-23T13:18:36Z

thanks for the reviews :) merging once the CI is green (edit: and also after a final commit of a missing docstring)

hanouticelina · 2025-07-23T13:39:14Z

src/huggingface_hub/commands/jobs.py

+Usage:
+    # run a job
+    huggingface-cli jobs run image command


(nit) expanding the top level docstring to list the other jobs sub commands
(cc @Wauplin, since the PR has been merged, we can do that in the follow-up PR that will switch huggingface-cli jobs -> hf jobs)

Suggested change

Usage:

# run a job

huggingface-cli jobs run image command

Usage:

# run a job

huggingface-cli jobs run <image> <command>

# List running or completed jobs

huggingface-cli jobs ps [-a] [-f key=value] [--format TEMPLATE]

# Stream logs from a job

huggingface-cli jobs logs <job-id>

# Inspect detailed information about a job

huggingface-cli jobs inspect <job-id>

# Cancel a running job

huggingface-cli jobs cancel <job-id>

done in #3250

lhoestq added 3 commits July 10, 2025 17:16

jobs

4836c04

style

682a789

docs

af05c27

lhoestq added 4 commits July 10, 2025 17:55

mypy

3895c8e

style

3661cb7

minor

13f17c8

remove hfjobs mentions

5e99d64

lhoestq commented Jul 10, 2025

View reviewed changes

src/huggingface_hub/commands/jobs/_cli_utils.py Outdated Show resolved Hide resolved

lhoestq mentioned this pull request Jul 10, 2025

Add minimal hfjobs uv run command lhoestq/hfjobs#8

Merged

lhoestq added 2 commits July 11, 2025 14:47

add huggingface-cli jobs uv commands

7efe998

add some uv options

ab8511e

lhoestq marked this pull request as ready for review July 11, 2025 14:04

lhoestq requested review from Wauplin, davanstrien and hanouticelina July 11, 2025 14:04

hanouticelina self-assigned this Jul 11, 2025

lhoestq commented Jul 11, 2025

View reviewed changes

setup.py Outdated Show resolved Hide resolved

lhoestq added 2 commits July 11, 2025 16:48

add test

3c00292

fix for 3.8

3136ef4

julien-c reviewed Jul 12, 2025

View reviewed changes

src/huggingface_hub/commands/jobs/uv.py Outdated Show resolved Hide resolved

julien-c reviewed Jul 12, 2025

View reviewed changes

Update src/huggingface_hub/commands/jobs/uv.py

9fc3c78

Co-authored-by: Julien Chaumond <[email protected]>

hanouticelina reviewed Jul 15, 2025

View reviewed changes

lhoestq added 3 commits July 16, 2025 23:58

move to HfApi

fd926b5

minor

1bf5f66

more comments

aefb493

lhoestq added 4 commits July 22, 2025 18:15

add missing methods at top level

76588ef

add docs

63dd90f

uv script url as env, not secret

bfd326a

rename docs

c9ab2f1

lhoestq added 3 commits July 22, 2025 18:28

update test

cf59dca

again

da1d40d

improve docs

334d831

Wauplin and others added 2 commits July 23, 2025 14:08

Merge branch 'main' into jobs

028d32a

add image arg to run_uv_job

fed7195

Wauplin added 2 commits July 23, 2025 14:41

List flavors from SpaceHardware

eaaa6a1

Merge branch 'jobs' of github.com:huggingface/huggingface_hub into jobs

c7660d7

Wauplin approved these changes Jul 23, 2025

View reviewed changes

lhoestq and others added 2 commits July 23, 2025 14:54

add to overview

af0e9fb

remove zero GPU from flavors

e6043ae

lhoestq added 3 commits July 23, 2025 15:11

add JobInfo etc. from _jobs_api in top level __init__

c444391

add package_reference doc page

ea6579a

minor - link JobInfo in docs

3f6a2f7

Wauplin approved these changes Jul 23, 2025

View reviewed changes

JobInfo docstring

3e049db

lhoestq merged commit 6e31992 into main Jul 23, 2025
23 of 25 checks passed

lhoestq deleted the jobs branch July 23, 2025 13:37

hanouticelina reviewed Jul 23, 2025

View reviewed changes

Wauplin mentioned this pull request Jul 23, 2025

Rename huggingface-cli jobs to hf jobs #3250

Merged

-Usage:
-    # run a job
-    huggingface-cli jobs run image command
+Usage:
+    # run a job
+    huggingface-cli jobs run <image> <command>
+    # List running or completed jobs
+    huggingface-cli jobs ps [-a] [-f key=value] [--format TEMPLATE]
+    # Stream logs from a job
+    huggingface-cli jobs logs <job-id>
+    # Inspect detailed information about a job
+    huggingface-cli jobs inspect <job-id>
+    # Cancel a running job
+    huggingface-cli jobs cancel <job-id>

[Jobs] Add huggingface-cli jobs commands #3211

[Jobs] Add huggingface-cli jobs commands #3211

Uh oh!

Conversation

lhoestq commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jul 10, 2025

Uh oh!

Uh oh!

lhoestq commented Jul 11, 2025

Uh oh!

Uh oh!

Uh oh!

julien-c Jul 12, 2025

Choose a reason for hiding this comment

Uh oh!

davanstrien Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davanstrien Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wauplin commented Jul 15, 2025

Uh oh!

lhoestq commented Jul 15, 2025

Uh oh!

hanouticelina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lhoestq commented Jul 22, 2025

Uh oh!

davanstrien commented Jul 23, 2025

Uh oh!

lhoestq commented Jul 23, 2025

Uh oh!

Wauplin commented Jul 23, 2025

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

julien-c commented Jul 23, 2025

Uh oh!

Wauplin commented Jul 23, 2025

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

lhoestq commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

hanouticelina Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Wauplin Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Wauplin Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

[Jobs] Add `huggingface-cli jobs` commands #3211

[Jobs] Add `huggingface-cli jobs` commands #3211

lhoestq commented Jul 10, 2025 •

edited

Loading

davanstrien Jul 14, 2025 •

edited

Loading

davanstrien Jul 14, 2025 •

edited

Loading

lhoestq commented Jul 23, 2025 •

edited

Loading