Promote benchmarks to library API (#6564)

# Description

Promotes the benchmark framework from `isaaclab.test.benchmark` to the
public `isaaclab.benchmark` package. Runtime, startup, training, play,
and RL backend workflows become library entrypoints with typed
in-process dispatch.

The transitional import and script shims were removed from this PR to
keep the move reviewable and leave one implementation for each workflow.
This is an intentional breaking change: callers must replace
`isaaclab.test.benchmark` imports with `isaaclab.benchmark` and use
`isaaclab benchmark` instead of the former runtime, startup, training,
and play scripts.

## Highlights

- Adds typed runtime, startup, training, and play request objects and
`run_benchmark` convenience APIs.
- Returns workflow-specific result types such as
`BenchmarkResult[RuntimeBundle]`.
- Adds in-process `isaaclab benchmark` CLI dispatch.
- Moves the supported benchmark workflows and RL backend adapters into
the package.
- Removes the duplicate `isaaclab.test.benchmark` namespace and
standalone workflow forwarding scripts.
- Restores process-global Torch, SKRL, and RL Games state and closes
environments safely across repeated calls.
- Rejects schema finalization without an attached typed bundle instead
of reporting a nonexistent output.
- Carries forward #6474 validation: positive workloads, non-negative
warm-up counts, one default train/play warm-up step, and
`--measure_sync_step`.
- Adds public API documentation, migration guidance, tests, and
changelog fragments.

## Dependencies

- Direct dependency: #6474.
- Satisfied prerequisite: #6553, already merged, provides reusable RL
train/play entrypoints.

## Breaking migration

- `from isaaclab.test.benchmark ...` becomes `from isaaclab.benchmark
...`.
- `scripts/benchmarks/{runtime,startup,training,play}.py` becomes
`isaaclab benchmark {runtime,startup,training,play}`.
- Per-library train/play scripts become the unified training/play
commands with `--rl_library`.
- Independent camera, loading, Hydra, and other standalone benchmarks
remain unchanged.

## Type of change

- New feature
- Breaking API migration
- Documentation update

## Validation

- Focused benchmark API, CLI, parser, adapter, and early-stop suite: 265
passed, 4 simulator smoke cases deselected.
- Focused public API and CLI entrypoint suite: 26 passed.
- `./isaaclab.sh -f`: all applicable hooks pass. The conflict-marker
hook was skipped because current `develop` contains literal `=======`
headings in three third-party license files.
- `tools/changelog/cli.py check develop`: passed.
- Runtime/startup smoke tests now invoke `uv run isaaclab benchmark`;
the local clean-`uv` bootstrap exceeded the validation window while
downloading the CUDA environment, so CI remains the end-to-end CLI
confirmation.

## Checklist

- [x] I have read and understood the contribution guidelines
- [x] I have run the pre-commit checks
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I have added tests for the public API and canonical CLI paths
- [x] I have added the required changelog fragments
- [x] I have documented the intentional breaking migration
- [x] My name already exists in `CONTRIBUTORS.md`

---------

Co-authored-by: Mustafa Haiderbhai <mhaiderbhai@nvidia.com>

Author: AntoineRichard

docs/conf.py

@@ -259,6 +259,7 @@ def _read_pinned_versions() -> dict:
     "h5py",
     "hid",
     "prettytable",
+    "psutil",
     "tqdm",
     "tensordict",
     "trimesh",

docs/source/api/index.rst

@@ -16,6 +16,7 @@ The following modules are available in the ``isaaclab`` extension:
    app
    actuators
    assets
+   benchmark
    cloner
    controllers
    devices

docs/source/api/lab/isaaclab.benchmark.rst

@@ -0,0 +1,110 @@
+isaaclab.benchmark
+==================
+
+.. automodule:: isaaclab.benchmark
+
+   .. rubric:: Request and result classes
+
+   .. autosummary::
+
+      BenchmarkLauncherConfig
+      BenchmarkOutputConfig
+      BenchmarkPlayRequest
+      BenchmarkResult
+      BenchmarkRuntimeRequest
+      BenchmarkStartupRequest
+      BenchmarkTrainingRequest
+
+   .. rubric:: Workflow functions
+
+   .. autosummary::
+
+      run_benchmark
+      run_play_benchmark
+      run_runtime_benchmark
+      run_startup_benchmark
+      run_training_benchmark
+
+   .. rubric:: Benchmark framework
+
+   .. autosummary::
+
+      BaseIsaacLabBenchmark
+      BenchmarkMonitor
+      MethodBenchmarkDefinition
+      MethodBenchmarkRunner
+      MethodBenchmarkRunnerConfig
+      PlayBundle
+      RuntimeBundle
+      StartupBundle
+      TrainingBundle
+
+
+.. currentmodule:: isaaclab.benchmark
+
+Request and Result Classes
+--------------------------
+
+.. autoclass:: BenchmarkLauncherConfig
+   :members:
+
+.. autoclass:: BenchmarkOutputConfig
+   :members:
+
+.. autoclass:: BenchmarkPlayRequest
+   :members:
+
+.. autoclass:: BenchmarkResult
+   :members:
+
+.. autoclass:: BenchmarkRuntimeRequest
+   :members:
+
+.. autoclass:: BenchmarkStartupRequest
+   :members:
+
+.. autoclass:: BenchmarkTrainingRequest
+   :members:
+
+Workflow Functions
+------------------
+
+.. autofunction:: run_benchmark
+
+.. autofunction:: run_play_benchmark
+
+.. autofunction:: run_runtime_benchmark
+
+.. autofunction:: run_startup_benchmark
+
+.. autofunction:: run_training_benchmark
+
+Benchmark Framework
+-------------------
+
+.. autoclass:: BaseIsaacLabBenchmark
+   :members:
+
+.. autoclass:: BenchmarkMonitor
+   :members:
+
+.. autoclass:: MethodBenchmarkDefinition
+   :members:
+
+.. autoclass:: MethodBenchmarkRunner
+   :members:
+
+.. autoclass:: MethodBenchmarkRunnerConfig
+   :members:
+
+.. autoclass:: PlayBundle
+   :members:
+
+.. autoclass:: RuntimeBundle
+   :members:
+
+.. autoclass:: StartupBundle
+   :members:
+
+.. autoclass:: TrainingBundle
+   :members:

docs/source/migration/migrating_to_isaaclab_3-0.rst

@@ -958,7 +958,7 @@ Here's a complete example showing how to update your code:
 
 
 Quaternion Format
-~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~
 
 **The quaternion format changed from WXYZ to XYZW.**
 
@@ -1946,36 +1946,41 @@ directly in your code, update your configuration:
    )
 
 
-Benchmark Scripts
-~~~~~~~~~~~~~~~~~
+Benchmark Workflows
+~~~~~~~~~~~~~~~~~~~
 
 Isaac Lab 3.0 consolidates the per-backend environment benchmark entry points and their
-wrapper shell runners into a small set of unified, backend-agnostic scripts. The physics
+wrapper shell runners into library-owned, backend-agnostic workflows. The physics
 backend is now selected at launch time through the ``presets=`` system — the same pattern
 used for environment configurations (see "Multi-Backend Support: PresetCfg Pattern" above)
 — rather than by choosing a backend-specific script.
 
 What Changed
 ------------
 
-The standalone environment benchmark entry-point scripts have been removed and replaced by
-unified scripts:
+The environment benchmark entry points are now exposed through ``isaaclab benchmark`` and
+the typed :mod:`isaaclab.benchmark` Python API:
 
-* ``runtime.py`` — steps an environment with random actions (no policy) and emits a
+* ``isaaclab benchmark runtime`` — steps an environment with random actions (no policy) and emits a
   ``RuntimeBundle``.
-* ``training.py`` — dispatches a real training run for the RL library selected with
+* ``isaaclab benchmark training`` — dispatches a real training run for the RL library selected with
   ``--rl_library`` and emits a ``TrainingBundle``.
-* ``startup.py`` — profiles the five startup phases (``app_launch``, ``python_imports``,
+* ``isaaclab benchmark startup`` — profiles the five startup phases (``app_launch``, ``python_imports``,
   ``task_config``, ``env_creation``, ``first_step``) with ``cProfile`` and emits a
   ``StartupBundle``.
-* ``play.py`` — **new in 3.0** — loads a trained checkpoint and benchmarks policy inference
+* ``isaaclab benchmark play`` — **new in 3.0** — loads a trained checkpoint and benchmarks policy inference
   for the RL library selected with ``--rl_library``, emitting a ``PlayBundle`` (inference
   throughput plus the policy's reward, episode length, and success rate). It consumes the
-  checkpoints produced by ``training.py``; 2.x had no per-backend play benchmark.
+  checkpoints produced by the training workflow; 2.x had no per-backend play benchmark.
 
 The wrapper shell runners that drove these benchmarks — ``run_non_rl_benchmarks.sh`` and
 ``run_training_benchmarks.sh`` — were removed as well; their behavior is now expressed
-directly through script arguments and ``presets=`` tokens.
+directly through command arguments and ``presets=`` tokens.
+
+The benchmark framework itself moved from :mod:`isaaclab.test.benchmark` to
+:mod:`isaaclab.benchmark`. The old namespace and the transitional runtime, startup,
+training, and play scripts were removed; update imports and invocations to the public
+module and unified command.
 
 .. note::
 
@@ -2001,30 +2006,30 @@ Map each old invocation to its replacement:
    * - Isaac Lab 2.x
      - Isaac Lab 3.0
    * - ``benchmark_non_rl.py``
-     - ``runtime.py`` (no ``--rl_library`` dispatch)
+     - ``isaaclab benchmark runtime`` (no ``--rl_library`` dispatch)
    * - ``benchmark_startup.py``
-     - ``startup.py``
+     - ``isaaclab benchmark startup``
    * - ``benchmark_rsl_rl.py``
-     - ``training.py --rl_library rsl_rl``
+     - ``isaaclab benchmark training --rl_library rsl_rl``
    * - ``benchmark_rlgames.py``
-     - ``training.py --rl_library rl_games``
+     - ``isaaclab benchmark training --rl_library rl_games``
    * - *(newly supported)*
-     - ``training.py --rl_library skrl``
+     - ``isaaclab benchmark training --rl_library skrl``
    * - *(newly supported)*
-     - ``training.py --rl_library sb3``
+     - ``isaaclab benchmark training --rl_library sb3``
    * - *(newly supported)*
-     - ``play.py --rl_library {rsl_rl,rl_games,skrl,sb3}``
+     - ``isaaclab benchmark play --rl_library {rsl_rl,rl_games,skrl,sb3}``
 
 SKRL and Stable-Baselines3 had no dedicated benchmark script in 2.x; both are now supported
-through the same ``--rl_library`` dispatch on ``training.py``. ``play.py`` is likewise new in
-3.0: it benchmarks inference of a checkpoint trained by ``training.py`` for any of the four
+through the same ``--rl_library`` dispatch on ``isaaclab benchmark training``. ``isaaclab benchmark play`` is likewise new in
+3.0: it benchmarks inference of a checkpoint trained by the training workflow for any of the four
 RL libraries.
 
 Running Benchmarks
 ------------------
 
 The physics (and rendering) backend is selected with Hydra preset tokens — ``presets=``,
-exactly as for ``train.py``. There is no ``--physics`` or ``--render`` flag; pass
+exactly as for the training workflow. There is no ``--physics`` or ``--render`` flag; pass
 ``presets=physx``, ``presets=newton_mjwarp``, etc. to choose the backend.
 
 **Before (Isaac Lab 2.x):**
@@ -2061,47 +2066,23 @@ exactly as for ``train.py``. There is no ``--physics`` or ``--render`` flag; pas
 
 **After (Isaac Lab 3.0):**
 
-.. tab-set::
-
-   .. tab-item:: uv (Recommended)
-
-      .. code-block:: bash
-
-         # Non-RL (random-action) runtime benchmark — PhysX (default)
-         uv run python scripts/benchmarks/runtime.py --task Isaac-Cartpole-Direct
-
-         # Same benchmark on Newton/MJWarp — select the backend via presets=
-         uv run python scripts/benchmarks/runtime.py --task Isaac-Cartpole-Direct presets=newton_mjwarp
-
-         # Training benchmark — choose the RL library with --rl_library
-         uv run python scripts/benchmarks/training.py --task Isaac-Cartpole-Direct --rl_library rsl_rl
-         uv run python scripts/benchmarks/training.py --task Isaac-Cartpole-Direct --rl_library skrl presets=newton_mjwarp
-
-         # Play (inference) benchmark — loads a checkpoint produced by training.py
-         uv run python scripts/benchmarks/play.py --task Isaac-Cartpole-Direct --rl_library rsl_rl --checkpoint /path/to/model.pt
-
-         # Startup profiling
-         uv run python scripts/benchmarks/startup.py --task Isaac-Cartpole-Direct presets=newton_mjwarp
-
-   .. tab-item:: isaaclab.sh / isaaclab.bat
-
-      .. code-block:: bash
+.. code-block:: bash
 
-         # Non-RL (random-action) runtime benchmark — PhysX (default)
-         ./isaaclab.sh -p scripts/benchmarks/runtime.py --task Isaac-Cartpole-Direct
+   # Non-RL (random-action) runtime benchmark — PhysX (default)
+   uv run isaaclab benchmark runtime --task Isaac-Cartpole-Direct
 
-         # Same benchmark on Newton/MJWarp — select the backend via presets=
-         ./isaaclab.sh -p scripts/benchmarks/runtime.py --task Isaac-Cartpole-Direct presets=newton_mjwarp
+   # Same benchmark on Newton/MJWarp — select the backend via presets=
+   uv run isaaclab benchmark runtime --task Isaac-Cartpole-Direct presets=newton_mjwarp
 
-         # Training benchmark — choose the RL library with --rl_library
-         ./isaaclab.sh -p scripts/benchmarks/training.py --task Isaac-Cartpole-Direct --rl_library rsl_rl
-         ./isaaclab.sh -p scripts/benchmarks/training.py --task Isaac-Cartpole-Direct --rl_library skrl presets=newton_mjwarp
+   # Training benchmark — choose the RL library with --rl_library
+   uv run isaaclab benchmark training --task Isaac-Cartpole-Direct --rl_library rsl_rl
+   uv run isaaclab benchmark training --task Isaac-Cartpole-Direct --rl_library skrl presets=newton_mjwarp
 
-         # Play (inference) benchmark — loads a checkpoint produced by training.py
-         ./isaaclab.sh -p scripts/benchmarks/play.py --task Isaac-Cartpole-Direct --rl_library rsl_rl --checkpoint /path/to/model.pt
+   # Play (inference) benchmark — loads a checkpoint produced by training
+   uv run isaaclab benchmark play --task Isaac-Cartpole-Direct --rl_library rsl_rl --checkpoint /path/to/model.pt
 
-         # Startup profiling
-         ./isaaclab.sh -p scripts/benchmarks/startup.py --task Isaac-Cartpole-Direct presets=newton_mjwarp
+   # Startup profiling
+   uv run isaaclab benchmark startup --task Isaac-Cartpole-Direct presets=newton_mjwarp
 
 Output Format
 -------------
@@ -2112,46 +2093,37 @@ comma-separated list to emit several formats at once. Supported values are ``sch
 ``omniperf``, ``osmo``, ``json``, and ``summary`` (legacy long-form aliases such as
 ``OmniPerfKPIFile`` are still accepted).
 
-.. tab-set::
-
-   .. tab-item:: uv (Recommended)
-
-      .. code-block:: bash
-
-         # Emit the typed schema bundle and an OmniPerf KPI file in one run
-         uv run python scripts/benchmarks/runtime.py --task Isaac-Cartpole-Direct \
-             --benchmark_formatter schema,omniperf
-
-   .. tab-item:: isaaclab.sh / isaaclab.bat
-
-      .. code-block:: bash
+.. code-block:: bash
 
-         # Emit the typed schema bundle and an OmniPerf KPI file in one run
-         ./isaaclab.sh -p scripts/benchmarks/runtime.py --task Isaac-Cartpole-Direct \
-             --benchmark_formatter schema,omniperf
+   # Emit the typed schema bundle and an OmniPerf KPI file in one run
+   uv run isaaclab benchmark runtime --task Isaac-Cartpole-Direct \
+       --benchmark_formatter schema,omniperf
 
 Migration Steps
 ---------------
 
 If you have custom benchmark scripts or CI based on Isaac Lab 2.x:
 
-1. **Replace the old entry points** — swap ``benchmark_non_rl.py`` for ``runtime.py``,
-   ``benchmark_startup.py`` for ``startup.py``, and the per-library training scripts for
-   ``training.py --rl_library <lib>``.
+1. **Replace the old entry points** — swap ``benchmark_non_rl.py`` for ``isaaclab benchmark runtime``,
+   ``benchmark_startup.py`` for ``isaaclab benchmark startup``, and the per-library training scripts for
+   ``isaaclab benchmark training --rl_library <lib>``.
+
+2. **Update benchmark imports** — replace ``isaaclab.test.benchmark`` with
+   ``isaaclab.benchmark``. The old namespace is no longer available.
 
-2. **Drop the wrapper runners** — ``run_non_rl_benchmarks.sh`` and
+3. **Drop the wrapper runners** — ``run_non_rl_benchmarks.sh`` and
    ``run_training_benchmarks.sh`` no longer exist; express their behavior with script
    arguments and ``presets=`` tokens. ``run_physx_benchmarks.sh`` is also gone — invoke the
    PhysX micro-benchmarks under ``source/isaaclab_physx/benchmark/`` directly instead.
 
-3. **Select the backend with** ``presets=`` — replace any per-backend script choice with a
+4. **Select the backend with** ``presets=`` — replace any per-backend script choice with a
    ``presets=`` (and, if needed, rendering) token on a single unified script. Update custom
    benchmark configs to the ``PresetCfg`` pattern.
 
-4. **Pick the output format with** ``--benchmark_formatter`` — default ``schema``; pass a
+5. **Pick the output format with** ``--benchmark_formatter`` — default ``schema``; pass a
    comma-separated list for multiple formats.
 
-5. **Test both backends** — verify your benchmarks pass with ``presets=physx`` (default) and
+6. **Test both backends** — verify your benchmarks pass with ``presets=physx`` (default) and
    ``presets=newton_mjwarp``.
 
 For a complete guide to multi-backend support, see the "Multi-Backend Support: PresetCfg