Adds a render config to the simulation and tiledCamera limitations to the docs (#1246)

# Description

This change adds a render config to the simulation context and
highlights current limitations and workarounds for issues with the
TiledCamera class.

## Type of change

- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

Author: kellyguo11

.github/workflows/docs.yaml

@@ -2,6 +2,10 @@ name: Build & deploy docs
 
 on:
   push:
+    branches:
+      - main
+  pull_request:
+    types: [opened, synchronize, reopened]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}

docs/index.rst

@@ -98,7 +98,6 @@ Table of Contents
    source/features/multi_gpu
    source/features/tiled_rendering
    source/features/reproducibility
-   .. source/features/motion_generators
 
 .. toctree::
    :maxdepth: 1

docs/source/api/lab/omni.isaac.lab.sim.rst

@@ -19,6 +19,7 @@
     SimulationContext
     SimulationCfg
     PhysxCfg
+    RenderCfg
 
   .. rubric:: Functions
 
@@ -46,6 +47,11 @@ Simulation Configuration
   :show-inheritance:
   :exclude-members: __init__
 
+.. autoclass:: RenderCfg
+  :members:
+  :show-inheritance:
+  :exclude-members: __init__
+
 Simulation Context Builder
 --------------------------
 

docs/source/features/hydra.rst

@@ -89,8 +89,8 @@ Elements in dictionaries are handled as a parameters in the hierarchy. For examp
 
 .. literalinclude:: ../../../source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_env_cfg.py
     :language: python
-    :lines: 99-111
-    :emphasize-lines: 10
+    :lines: 90-114
+    :emphasize-lines: 11
 
 the ``position_range`` parameter can be modified with ``env.events.reset_cart_position.params.position_range="[-2.0, 2.0]"``.
 This example shows two noteworthy points:
@@ -112,7 +112,7 @@ For example, for the configuration of the Cartpole camera depth environment:
     :language: python
     :start-at: class CartpoleDepthCameraEnvCfg
     :end-at: tiled_camera.width
-    :emphasize-lines: 16
+    :emphasize-lines: 10, 15
 
 If the user were to modify the width of the camera, i.e. ``env.tiled_camera.width=128``, then the parameter
 ``env.observation_space=[80,128,1]`` must be updated and given as input as well.

docs/source/features/tiled_rendering.rst

@@ -8,7 +8,7 @@ Tiled-Camera Rendering
     This feature is only available from Isaac Sim version 4.2.0 onwards.
 
     Tiled rendering in combination with image processing networks require heavy memory resources, especially
-    at larger resolutions. We recommend running at 512 cameras in the scene on RTX 4090 GPUs or similar.
+    at larger resolutions. We recommend running 512 cameras in the scene on RTX 4090 GPUs or similar.
 
 
 Tiled rendering APIs provide a vectorized interface for collecting data from camera sensors.
@@ -129,3 +129,39 @@ Instance Segmentation
 - If ``colorize_instance_segmentation=True`` in the camera config, a 4-channel RGBA image will be returned with dimension (B, H, W, 4) and type ``torch.uint8``. The info ``idToLabels`` dictionary will be the mapping from color to USD prim path of that semantic entity. The info ``idToSemantics`` dictionary will be the mapping from color to semantic labels of that semantic entity.
 
 - If ``colorize_instance_segmentation=False``, a buffer of dimension (B, H, W, 1) of type ``torch.int32`` will be returned, containing the instance ID of each pixel. The info ``idToLabels`` dictionary will be the mapping from instance ID to USD prim path of that semantic entity. The info ``idToSemantics`` dictionary will be the mapping from instance ID to semantic labels of that semantic entity.
+
+
+Current Limitations
+-------------------
+
+Due to current limitations in the renderer, we can have only **one** :class:`~sensors.TiledCamera` instance in the scene.
+For use cases that require a setup with more than one camera, we can imitate the multi-camera behavior by moving the location
+of the camera in between render calls in a step.
+
+For example, in a stereo vision setup, the below snippet can be implemented:
+
+.. code-block:: python
+
+    # render image from "first" camera
+    camera_data_1 = self._tiled_camera.data.output["rgb"].clone() / 255.0
+    # update camera transform to the "second" camera location
+    self._tiled_camera.set_world_poses(
+        positions=pos,
+        orientations=rot,
+        convention="world"
+    )
+    # step the renderer
+    self.sim.render()
+    self._tiled_camera.update(0, force_recompute=True)
+    # render image from "second" camera
+    camera_data_2 = self._tiled_camera.data.output["rgb"].clone() / 255.0
+
+Note that this approach still limits the rendering resolution to be identical for all cameras. Currently, there is no workaround
+to achieve different resolution images using :class:`~sensors.TiledCamera`. The best approach is to use the largest resolution out of all of the
+desired resolutions and add additional scaling or cropping operations to the rendered output as a post-processing step.
+
+In addition, there may be visible quality differences when comparing render outputs of different numbers of environments.
+Currently, any combined resolution that has a width less than 265 pixels or height less than 265 will automatically switch
+to the DLAA anti-aliasing mode, which does not perform up-sampling during anti-aliasing. For resolutions larger than 265 in both
+width and height dimensions, we default to using the "performance" DLSS mode for anti-aliasing for performance benefits.
+Anti-aliasing modes and other rendering parameters can be specified in the :class:`~sim.RenderCfg`.

docs/source/overview/core-concepts/index.rst

@@ -9,3 +9,5 @@ This section we introduce core concepts in Isaac Lab.
 
   task_workflows
   actuators
+
+# motion_generators

source/extensions/omni.isaac.lab/config/extension.toml

@@ -1,7 +1,7 @@
 [package]
 
 # Note: Semantic Versioning is used: https://semver.org/
-version = "0.27.0"
+version = "0.27.1"
 
 # Description
 title = "Isaac Lab framework for Robot Learning"

source/extensions/omni.isaac.lab/docs/CHANGELOG.rst

@@ -1,6 +1,16 @@
 Changelog
 ---------
 
+0.27.1 (2024-10-20)
+~~~~~~~~~~~~~~~~~~~
+
+Added
+^^^^^
+
+* Added :class:`~omni.isaac.lab.sim.RenderCfg` and the attribute :attr:`~omni.isaac.lab.sim.SimulationCfg.render` for
+  specifying render related settings.
+
+
 0.27.0 (2024-10-14)
 ~~~~~~~~~~~~~~~~~~~
 

source/extensions/omni.isaac.lab/omni/isaac/lab/sim/__init__.py

@@ -28,7 +28,7 @@
 
 from .converters import *  # noqa: F401, F403
 from .schemas import *  # noqa: F401, F403
-from .simulation_cfg import PhysxCfg, SimulationCfg  # noqa: F401, F403
+from .simulation_cfg import PhysxCfg, RenderCfg, SimulationCfg  # noqa: F401, F403
 from .simulation_context import SimulationContext, build_simulation_context  # noqa: F401, F403
 from .spawners import *  # noqa: F401, F403
 from .utils import *  # noqa: F401, F403

source/extensions/omni.isaac.lab/omni/isaac/lab/sim/simulation_cfg.py

@@ -152,6 +152,54 @@ class PhysxCfg:
     """Size of particle contacts stream buffer allocated in pinned host memory. Default is 2 ** 20."""
 
 
+@configclass
+class RenderCfg:
+    """Configuration for Omniverse RTX Renderer.
+
+    These parameters are used to configure the Omniverse RTX Renderer.
+    For more information, see the `Omniverse RTX Renderer documentation`_.
+
+    .. _Omniverse RTX Renderer documentation: https://docs.omniverse.nvidia.com/materials-and-rendering/latest/rtx-renderer.html
+    """
+
+    enable_translucency: bool = False
+    """Enables translucency for specular transmissive surfaces such as glass at the cost of some performance. Default is False."""
+
+    enable_reflections: bool = False
+    """Enables reflections at the cost of some performance. Default is False."""
+
+    enable_global_illumination: bool = False
+    """Enables Diffused Global Illumination at the cost of some performance. Default is False."""
+
+    antialiasing_mode: Literal["Off", "FXAA", "DLSS", "TAA", "DLAA"] = "DLSS"
+    """Selects the anti-aliasing mode to use. Defaults to DLSS."""
+
+    enable_dlssg: bool = False
+    """"Enables the use of DLSS-G.
+        DLSS Frame Generation boosts performance by using AI to generate more frames.
+        DLSS analyzes sequential frames and motion data to create additional high quality frames.
+        This feature requires an Ada Lovelace architecture GPU.
+        Enabling this feature also enables additional thread-related activities, which can hurt performance.
+        Default is False."""
+
+    dlss_mode: Literal[0, 1, 2, 3] = 0
+    """For DLSS anti-aliasing, selects the performance/quality tradeoff mode.
+       Valid values are 0 (Performance), 1 (Balanced), 2 (Quality), or 3 (Auto). Default is 0."""
+
+    enable_direct_lighting: bool = True
+    """Enable direct light contributions from lights."""
+
+    samples_per_pixel: int = 1
+    """Defines the Direct Lighting samples per pixel.
+       Higher values increase the direct lighting quality at the cost of performance. Default is 1."""
+
+    enable_shadows: bool = True
+    """Enables shadows at the cost of performance. When disabled, lights will not cast shadows. Defaults to True."""
+
+    enable_ambient_occlusion: bool = False
+    """Enables ambient occlusion at the cost of some performance. Default is False."""
+
+
 @configclass
 class SimulationCfg:
     """Configuration for simulation physics."""
@@ -234,3 +282,6 @@ class SimulationCfg:
 
     The material is created at the path: ``{physics_prim_path}/defaultMaterial``.
     """
+
+    render: RenderCfg = RenderCfg()
+    """Render settings. Default is RenderCfg()."""