[benchmarking] Adds support for benchmarking pre-26.02 Curator releases#1431

Draft

rlratzel wants to merge 50 commits intoNVIDIA-NeMo:mainfrom

rlratzel:26.02-benchmark_2509_updates

Contributor

rlratzel commented Jan 26, 2026 •

edited

Loading

Adds initial updates to support benchmarking pre-26.02 Curator code:

Docker build script updates to use a specific Curator image instead of building from source
Using different RayClient API based on Curator version

Note to reviewers: this PR is in Draft until we know we want to add this functionality


          Adds initial updates to support benchmarking pre-26.02 Curator code: …

838b2a8

…Docker build script updates to use a specific Curator image instead of building from source, using different RayClient API based on Curator version. More changes are needed before support is complete.

Signed-off-by: rlratzel <rratzel@nvidia.com>

copy-pr-bot bot commented Jan 26, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

rlratzel changed the title ~~Adds support for benchmarking pre-26.02 Curator releases~~ [benchmarking] Adds support for benchmarking pre-26.02 Curator releases

rlratzel and others added 27 commits

January 26, 2026 15:51


          Copies TaskPerfUtils static methods to benchmark runner for use with …

f3db50f

…pre-26.02 Curator objects, updates audio benchmark to handle Path object.

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Adds ability to use local benchmarking tools in a container without m…

3113d6c

…ounting the entire curator source tree.

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

01cee82

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Changes option to skip Curator image build to be consistent with use …

67ca4c5

…of env vars.

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Copies config dir needed by newer benchmarks if not present in older …

5443d87

…images.

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

f3331ee

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Adds option to also skip pulling the curator image.

99b3cdd

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Forces an update to the benchmark tools in the image to ensure correc…

e58745c

…t version, increases various timeouts

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Increases various timeouts slightly more just in case

bc71276

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

61f259a

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Adds workarounds for running on legacy versions of Curator

00d0ea1

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Changes for backward compat with 25.09, moves Task helpers to scripts…

b0c047f

… utils to be shared by framework and scripts.

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Makes additional changes for 25.09 compat

49e4960

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Update scripts_task type check to allow lists for aggregate_task_metr…

7b95943

…ics_for_legacy

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

3ffff62

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Updates the num clips generated check for video captioning.

66d48a5

Signed-off-by: rlratzel <rratzel@nvidia.com>


          ci: Optimize docker layer and uv with no cache (NVIDIA-NeMo#1444)

aacc09a

* Optimize docker layer and uv with no cache

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>

* Add missing slash

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>

* Add comments to dockerfile

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>

---------

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>


          git mess

fa62060

Signed-off-by: Praateek <praateekm@gmail.com>


          Merge pull request #1 from praateekmahajan/praateek/2509-benchmarking

61f04b1

Make changes for dedup pipelines to run on 25.09 and 26.02


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

24b7f75

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

80ca1c2

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

d1fb505

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

66cf5c8

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Adds support for constructing legacy ImageReaderStage classes

f477156

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

be0a824

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Increases timeout for image benchmark

12a1f8d

Signed-off-by: rlratzel <rratzel@nvidia.com>

rlratzel added 18 commits

February 6, 2026 13:38


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

a76849b

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream' into 26.02-benchmark_2509_upd…

3b1353b

…ates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Updates Slack sink to use Slack SDK, post as bot to create threads an…

d092fc8

…d live updates.

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Refactors helper functions into base class, adds better fallback text…

f1b7f2b

…, adds unique emojis, updates YAML file with new options

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Renamed var for clarity.

bc2387e

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Replaces old references to webhook_url with channel_id

61db095

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_sla…

7b7a8cf

…ck_sink_update

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

c0306dd

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Updates run.sh and README for new Slack env vars.

e7b2c60

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Updates type annotations to allow None.

a1778b0

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Updates type annotations to allow None.

2848b10

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_sla…

6bdee26

…ck_sink_update

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

1bcb9a9

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge branch '26.02-benchmark_slack_sink_update' into 26.02-benchmark…

8c9eeda

…_2509_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Removes videos_model_weights in favor of downloading latest weights.

a00c00c

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

2ed5d5c

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

088e708

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>

copy-pr-bot bot commented Feb 12, 2026

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

rlratzel added 4 commits

February 17, 2026 17:04


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

eb7b851

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Adds Nvidia driver env var for video benchmarks

98b97b5

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

fe43f73

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>


          Merge remote-tracking branch 'upstream/main' into 26.02-benchmark_250…

0f04ea3

…9_updates

Signed-off-by: rlratzel <rratzel@nvidia.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

praateekmahajan Awaiting requested review from praateekmahajan praateekmahajan will be requested when the pull request is marked ready for review praateekmahajan is a code owner

sarahyurick Awaiting requested review from sarahyurick sarahyurick will be requested when the pull request is marked ready for review sarahyurick is a code owner

ayushdg Awaiting requested review from ayushdg ayushdg will be requested when the pull request is marked ready for review ayushdg is a code owner

At least 1 approving review is required to merge this pull request.

Labels

None yet