Expose a handle for the Python viewer.

This change also requires user scripts to explicitly synchronize changes to physics state to the viewer. The Simulate class was reconfigured so that certain UI events are handled during this sync operation, outside of the render loop on the main thread. These correspond to operations that require access to the full mjModel/mjData.

To support other, more interactive operations (e.g. camera movements), a new mjvSceneState struct is introduced which captures only the portion of the physics state required for scene re-rendering. The mjvSceneState is updated from mjModel/mjData during the viewer sync operation, and is significantly cheaper than a full mj_copyModel and mj_copyData.

Fixes #796

PiperOrigin-RevId: 525723636
Change-Id: Id08d0210a2c067d5afe85e2bf104f276aeddd75e

Author: saran-t

doc/APIreference/APItypes.rst

@@ -814,6 +814,17 @@ This structure contains everything needed to render the 3D scene in OpenGL.
 .. mujoco-include:: mjvScene
 
 
+.. _mjvSceneState:
+
+mjvSceneState
+~~~~~~~~~~~~~
+
+This structure contains the portions of :ref:`mjModel` and :ref:`mjData` that are required for
+various ``mjv_*`` functions.
+
+.. mujoco-include:: mjvScene
+
+
 .. _mjvFigure:
 
 mjvFigure

doc/APIreference/functions.rst

@@ -1332,6 +1332,15 @@ mjv_moveCamera
 
 Move camera with mouse; action is mjtMouse.
 
+.. _mjv_moveCameraFromState:
+
+mjv_moveCameraFromState
+~~~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_moveCameraFromState
+
+Move camera with mouse given a scene state; action is mjtMouse.
+
 .. _mjv_movePerturb:
 
 mjv_movePerturb
@@ -1341,6 +1350,15 @@ mjv_movePerturb
 
 Move perturb object with mouse; action is mjtMouse.
 
+.. _mjv_movePerturbFromState:
+
+mjv_movePerturbFromState
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_movePerturbFromState
+
+Move perturb object with mouse given a scene state; action is mjtMouse.
+
 .. _mjv_moveModel:
 
 mjv_moveModel
@@ -1483,6 +1501,51 @@ mjv_updateScene
 
 Update entire scene given model state.
 
+.. _mjv_updateSceneFromState:
+
+mjv_updateSceneFromState
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_updateSceneFromState
+
+Update entire scene from a scene state, return the number of new mjWARN_VGEOMFULL warnings.
+
+.. _mjv_defaultSceneState:
+
+mjv_defaultSceneState
+~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_defaultSceneState
+
+Set default scene state.
+
+.. _mjv_makeSceneState:
+
+mjv_makeSceneState
+~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_makeSceneState
+
+Allocate resources and initialize a scene state object.
+
+.. _mjv_freeSceneState:
+
+mjv_freeSceneState
+~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_freeSceneState
+
+Free scene state.
+
+.. _mjv_updateSceneState:
+
+mjv_updateSceneState
+~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_updateSceneState
+
+Update a scene state from model and data.
+
 .. _mjv_addGeoms:
 
 mjv_addGeoms
@@ -1572,6 +1635,15 @@ mjr_freeContext
 
 Free resources in custom OpenGL context, set to default.
 
+.. _mjr_resizeOffscreen:
+
+mjr_resizeOffscreen
+~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjr_resizeOffscreen
+
+Resize offscreen buffers.
+
 .. _mjr_uploadTexture:
 
 mjr_uploadTexture

doc/changelog.rst

@@ -35,6 +35,10 @@ Python bindings
   state concurrently with the internal ``mj_forward``, resulting in e.g.
   `MuJoCo stack overflow error <https://github.com/deepmind/mujoco/issues/783>`_
   or `segmentation fault <https://github.com/deepmind/mujoco/issues/790>`_.
+- The ``viewer.launch_passive`` function now returns a handle which can be used to interact with the viewer. The passive
+  viewer now also requires an explicit call to ``sync`` on its handle to pick up any update to the physics state. This
+  is to avoid race conditions that can result in visual artifacts. See :ref:`documentation<PyViewer>` for details.
+- The ``viewer.launch_repl`` function has been removed since its functionality is superceded by ``launch_passive``.
 - Added a small number of missing struct fields discovered through the new ``introspect`` metadata.
 
 Bug fixes
@@ -102,7 +106,7 @@ Python bindings
 #.  Added ``viewer.launch_passive`` which launches the interactive viewer in a passive, non-blocking mode. Calls to
     ``launch_passive`` return immediately, allowing user code to continue execution, with the viewer automatically
     reflecting any changes to the physics state. (Note that this functionality is currently in experimental/beta stage,
-    and is not yet described in our :ref:`viewer  documentation<PyViewer>`.)
+    and is not yet described in our :ref:`viewer documentation<PyViewer>`.)
 #.  Added the ``mjpython`` launcher for macOS, which is required for ``viewer.launch_passive`` to function there.
 #.  Removed ``efc_`` fields from joint indexers. Since the introduction of arena memory, these fields now have dynamic
     sizes that change between time steps depending on the number of active constraints, breaking strict correspondence