Aggregate open benchmarking work by AntoineRichard · Pull Request #6769 · isaac-sim/IsaacLab

AntoineRichard · 2026-07-28T21:47:58Z

Summary

Integration draft aggregating the currently open benchmarking pull requests after rebasing them onto develop, which now contains #6564. The individual pull requests remain the intended review and merge units; this draft exists to exercise their combined behavior and CI.

Included pull requests

Fix articulation data micro-benchmarks #6565 — Fix articulation data micro-benchmarks
Add OVPhysX asset micro-benchmarks #6566 — Add OVPhysX asset micro-benchmarks
Add OVPhysX sensor micro-benchmarks #6567 — Add OVPhysX sensor micro-benchmarks
Add Newton sensor micro-benchmarks #6568 — Add Newton sensor micro-benchmarks
Document micro-benchmark workflows #6587 — Document micro-benchmark workflows
Rewrite benchmark user guide #6595 — Rewrite benchmark user guide
Rewrite advanced benchmark framework guide #6627 — Rewrite advanced benchmark framework guide
Unify benchmark documentation journey #6629 — Unify benchmark documentation journey
Success series in benchmark schema #6624 — Success series in benchmark schema
Restore RL entrypoint backend state #6631 — Restore RL entrypoint backend state
Standardize benchmark warm-up steps #6764 — Standardize benchmark warm-up steps

Stack

The micro-benchmark and documentation work is linearized as:

#6565 -> #6566 -> #6567 -> #6568 -> #6587 -> #6595 -> #6627 -> #6629

The aggregate then merges the independent #6624, #6631, and #6764 tips. Historical plan/spec commits were excluded from the rewritten branches.

Integration notes

The success-rate learning curve advances the benchmark bundle schema from 1.2 to 1.3.
The combined user guide keeps the rewritten workflow structure while using num_steps and warmup_steps consistently.
The documentation-navigation merge includes only its three benchmark documentation files; unrelated stale develop history was not carried forward.

Verification

./isaaclab.sh -f
Combined benchmark, smoke, mock-interface, and RL entrypoint tests: 349 passed, 1 skipped, 4 deselected

Keep mock state buffers stable so timed samples measure accessor work instead of random-data allocation or mock-side copies. Initialize Newton dynamics metadata and make unsupported or deprecated properties explicit.

Benchmark articulation, rigid-object, and collection wrapper and data paths using the optional kitless OVPhysX runtime. Keep mock storage work outside timed operations so results reflect Isaac Lab overhead.

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

Share synchronized latency sampling across PhysX and OVPhysX sensor workloads. Emit summary and JSON results with observer-floor, submission, completion, and native-read statistics so runs are directly comparable and failures remain visible.

Use the shared synchronized latency runner so Newton sensor results include consistent statistics, observer cost, metadata, and machine-readable output across backends.

Explain why isolated asset and sensor benchmarks exist, how to run each backend workload, and how to interpret and compare their outputs.

Keep the designated-workstation requirement public without exposing the private local SSH helper.

Lead with the supported typed Python workflows, document verified request and result contracts, and provide an executable lower-level extension example. Align API reference coverage and testing navigation with the three-part benchmark journey.

Connect the benchmark user, framework, and micro-benchmark guides through one coherent documentation path while preserving their distinct scopes.

Scope Torch and SKRL global backend settings to reusable in-process training and playback calls. Preserve attributes that were initially absent and add failure-path regressions for RSL-RL and SKRL.

Move shared state guards into the common entrypoint utilities and make RSL-RL's Torch backend policy explicit at the call site. Scope SKRL playback preservation to JAX calls and cover exceptional RSL-RL cleanup.

Use step terminology across runtime and play interfaces. Record the excluded count in environment-step timing so schema and flat formatter consumers share one contract.

Advance the rebased benchmark bundle schema from 1.2 to 1.3 for the additive learning success-rate curve.

# Conflicts: # scripts/benchmarks/test/test_benchmark_smoke.py

# Conflicts: # docs/source/testing/benchmarks.rst

StafaH · 2026-07-28T23:12:54Z

@@ -125,27 +125,10 @@ CPU: Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz
 +-------------------------------------+-------------------+--------------+-------------------+--------------------+


We can probably remove this whole file at this point

StafaH · 2026-07-28T23:15:17Z

            learning = builders.build_learning(
                reward_series=log_data.get(desc.reward_tag, []),
                ep_length_series=log_data.get(desc.ep_length_tag, []),
+                success_rate_series=tracker.history if tracker is not None else None,


-> =tracker.history, if its none it'll get set to None

StafaH · 2026-07-28T23:17:29Z



+def _build_learning_curve(values: Sequence[float], ema_alpha: float, keep_series: bool) -> LearningCurve:
+    samples = list(values)


is there a concern that this would be some other type? If it is why is the type hint Sequence?

StafaH · 2026-07-28T23:20:14Z

+    return stats
+
+
+def _percentile(samples: Sequence[float], fraction: float) -> float:


Can we leverage numpy for doing our math here and elsewhere. numpy has percentiles, std_dev, and other strong helpers. The calculations can then be done in-place

StafaH · 2026-07-28T23:20:54Z

    environment_step_calls: int
    simulation_step_calls: int | None
    measurement_mode: Literal["host_return", "serialized_synchronized"]
+    warmup_steps: int = 0


is the default meant to be 0? I thought it was 1?

StafaH · 2026-07-28T23:29:50Z

+        wp.synchronize_device(sim.device)
+
+        # Time only sensor work. Physics steps happen outside the timed regions.
+        synchronize = partial(wp.synchronize_device, sim.device)


Not good to hide that you are using warp synchronize for a small refactor gain

StafaH · 2026-07-28T23:31:21Z

@@ -0,0 +1,201 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).


Does rigid body really need 3 different benchmark files. Code could be shared if they were together. I know they benchmark three different concepts but worth thinking if we can combine.

StafaH · 2026-07-28T23:33:29Z

+
+
+@configclass
+class RayCasterBenchmarkSceneCfg(InteractiveSceneCfg):


ray-plane is not that interesting of a benchmark. Similarly for the other benchmarks we might need to choose (and document) them a bit more carefully. Can be done in follow-up

StafaH · 2026-07-28T23:38:29Z

@@ -16,6 +16,8 @@
 from __future__ import annotations


Here and elsewhere, we should probably make these microbenchmarks callable from the main benchmarking command

uv run isaaclab microbenchmark --component contact_sensor physics=newton_mjwarp

Can be done in follow up or directly here, just leaving my brainstormed thoughts

StafaH · 2026-07-29T00:07:56Z

 AppLauncher.add_app_launcher_args(parser)
 args_cli = parser.parse_args()
+if args_cli.num_envs <= 0:
+    parser.error("--num_envs must be greater than zero")


type check in add_argument?

AntoineRichard added 28 commits July 28, 2026 23:33

Fix articulation data micro-benchmarks

4f2482e

Keep mock state buffers stable so timed samples measure accessor work instead of random-data allocation or mock-side copies. Initialize Newton dynamics metadata and make unsupported or deprecated properties explicit.

Add OVPhysX asset micro-benchmarks

e3eb0e2

Benchmark articulation, rigid-object, and collection wrapper and data paths using the optional kitless OVPhysX runtime. Keep mock storage work outside timed operations so results reflect Isaac Lab overhead.

Add OVPhysX sensor micro-benchmarks

f3a0987

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

Harden sensor micro-benchmark timing

8ffe3e0

Share synchronized latency sampling across PhysX and OVPhysX sensor workloads. Emit summary and JSON results with observer-floor, submission, completion, and native-read statistics so runs are directly comparable and failures remain visible.

Add Newton sensor micro-benchmarks

6ecdc8c

Standardize Newton sensor benchmarks

088147d

Use the shared synchronized latency runner so Newton sensor results include consistent statistics, observer cost, metadata, and machine-readable output across backends.

Document micro-benchmark workflows

e26db4c

Explain why isolated asset and sensor benchmarks exist, how to run each backend workload, and how to interpret and compare their outputs.

Clarify benchmark publication provenance

55ff4e2

Keep the designated-workstation requirement public without exposing the private local SSH helper.

Reorganize benchmark documentation

d5c93a9

Use standard benchmark output examples

89a40e2

Rewrite benchmark user guide

db2bd43

Explain startup iteration cost

0af66f8

Fix benchmark schema provenance

14c5264

Add benchmark workflow examples

f39c303

Rewrite advanced benchmark framework guide

63c8f81

Lead with the supported typed Python workflows, document verified request and result contracts, and provide an executable lower-level extension example. Align API reference coverage and testing navigation with the three-part benchmark journey.

Unify benchmark documentation navigation

caeda2c

Connect the benchmark user, framework, and micro-benchmark guides through one coherent documentation path while preserving their distinct scopes.

Add training success-rate curves

0f71c33

Record training success-rate series

63c5c82

Simplify benchmark success smoke checks

cd256b4

Use logged success series in bundles

e5240d7

Fix success tracker documentation

7242587

Restore RL entrypoint backend state

f0b30fe

Scope Torch and SKRL global backend settings to reusable in-process training and playback calls. Preserve attributes that were initially absent and add failure-path regressions for RSL-RL and SKRL.

Refine RL entrypoint state restoration

bb7cee1

Move shared state guards into the common entrypoint utilities and make RSL-RL's Torch backend policy explicit at the call site. Scope SKRL playback preservation to JAX calls and cover exceptional RSL-RL cleanup.

Standardize benchmark warm-up steps

3627310

Use step terminology across runtime and play interfaces. Record the excluded count in environment-step timing so schema and flat formatter consumers share one contract.

Bump success series schema version

15b1d9c

Advance the rebased benchmark bundle schema from 1.2 to 1.3 for the additive learning success-rate curve.

Merge benchmark success series

ca1a3fd

# Conflicts: # scripts/benchmarks/test/test_benchmark_smoke.py

Merge RL entrypoint state restoration

c304c58

Merge benchmark warm-up step standardization

112c255

# Conflicts: # docs/source/testing/benchmarks.rst

github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Jul 28, 2026

StafaH approved these changes Jul 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Aggregate open benchmarking work - #6769

Aggregate open benchmarking work#6769
AntoineRichard wants to merge 28 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/benchmarking-giga-stack

AntoineRichard commented Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 28, 2026

Uh oh!

StafaH Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -125,27 +125,10 @@ CPU: Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz
		+-------------------------------------+-------------------+--------------+-------------------+--------------------+



		def _build_learning_curve(values: Sequence[float], ema_alpha: float, keep_series: bool) -> LearningCurve:
		samples = list(values)

		return stats


		def _percentile(samples: Sequence[float], fraction: float) -> float:

		@@ -0,0 +1,201 @@
		# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).



		@configclass
		class RayCasterBenchmarkSceneCfg(InteractiveSceneCfg):

Uh oh!

Conversation

AntoineRichard commented Jul 28, 2026

Summary

Included pull requests

Stack

Integration notes

Verification

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants