Skip to content

Super 3.5 evals update 20260813 - #2549

Merged
bxyu-nvidia merged 8 commits into
mainfrom
bxyu/super3.5-evals-update-20260813
Aug 13, 2026
Merged

Super 3.5 evals update 20260813#2549
bxyu-nvidia merged 8 commits into
mainfrom
bxyu/super3.5-evals-update-20260813

Conversation

@bxyu-nvidia

@bxyu-nvidia bxyu-nvidia commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

  1. Split Super serving prefill/decode args: benchmarks/nemotron_3.5_super/vllm_configs/nemotron_3.5_super.sh, benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh
  2. Upgrade to Tomer's vLLM 0.27.1 container build script: benchmarks/nemotron_3.5_super/build-super-vl-evals-v0271-thin.sh, benchmarks/nemotron_3.5_super/build-super-vl-rl-v0251-thin.sh
  3. No longer downgrade to Python 3.12 since we don't need to match vLLM Python anymore due to not using Ray: benchmarks/nemotron_3.5_super/build_eval_container.sh, benchmarks/nemotron_3.5_super/downgrade_python.py
  4. Cleanup sbatch script: benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh
  5. AA-Omniscience benchmark uses HF token in prepare: benchmarks/omniscience/prepare.py
  6. Upgrade intermediate results aggregation script: scripts/print_aggregate_results.py
  7. Upgrade SWE Bench resources server and OpenCodeSandboxedAgent to use PTY: resources_servers/swebench/app.py, responses_api_agents/opencode_sandboxed_agent/app.py

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
@bxyu-nvidia
bxyu-nvidia merged commit 9eeeee5 into main Aug 13, 2026
16 checks passed
@bxyu-nvidia
bxyu-nvidia deleted the bxyu/super3.5-evals-update-20260813 branch August 13, 2026 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant