How to implement custom GUI elements and trajectory visualization in MuJoCo C API (for beginners)

[jheyeo222]

Intro

Hi!

I am an undergraduate student conducting research on force control. I'm relatively new to computer programming.

My setup

• MuJoCo version: 2.3.7
• OS: Windows 10
• API: C/C++

My question

My question

I've been watching Yuval Tassa's YouTube videos on MJPC and I'm trying to implement similar visualization features as shown in the screenshots.

**Specifically, I want to implement:

1. The GUI elements marked as "1" and "2" in images
2. The path/trajectory visualization marked as "3" and "4" and "5"

I've written basic MuJoCo code, but I can't figure out how to add these GUI and visualization features. As a coding beginner, I would really appreciate any tutorials, examples, or API documentation that could help me understand how to implement these GUI features using the MuJoCo 2.3.7 C API.

Minimal model and/or code that explain my question

If you encountered the issue in a complex model, please simplify it as much as possible (while still reproducing the issue).

Model:

minimal XML

<mujoco>
  <worldbody>
    <light pos="0 0 1"/>
    <geom type="sphere" size="1" rgba="1 0 0 1"/>
  </worldbody>
</mujoco>

Code:

import mujoco
import mediapy as media
model = mujoco.MjModel.from_xml_string(xml)
data = mujoco.MjData(model)

with mujoco.Renderer(model) as renderer:
  mujoco.mj_forward(model, data)
  renderer.update_scene(data)

  media.show_image(renderer.render())

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[v-r-a]

Hello, I am a MuJoCo user. This answers your second point.

I believe the first two snapshots are just sites being visualised. For adding decorative geometries--

If you write in C/C++, you can add decorative geometries by calling mjv_initgeom followed by mjv_connector. I made some custom functions for myself based on the code of the above functions. They might be helpful to you as well.

Example1: draw a sphere.

void drawSph(const mjModel *mm, mjData *dd, mjtNum pt[3], mjtNum radius, float rgba[], mjvScene *scene, const mjvOption *opt)
{
    // add a decorative geometry
    mjvGeom *mygeom;
    mjtNum sphsize[3] = {radius, 0, 0};
    mjtNum sphpos[3] = {0., 0., 0.};
    mjtNum myrot3x3[9] = {1., 0., 0., 0., 1., 0., 0., 0., 1.};

    mju_copy3(sphpos, pt);

    // one more geom to render
    scene->ngeom = scene->ngeom + 1;
    // mygeom now points to the last location in geoms buffer
    mygeom = scene->geoms + scene->ngeom - 1;
    // decor geom
    mygeom->objtype = mjOBJ_UNKNOWN;
    mygeom->objid = -1;
    mygeom->category = mjCAT_DECOR;
    mygeom->segid = scene->ngeom;

    // Add it to the scene
    mjv_initGeom(mygeom, mjGEOM_SPHERE, sphsize, sphpos, myrot3x3, rgba);
    mjv_addGeoms(mm, dd, opt, NULL, mjCAT_DECOR, scene);
}

One should also specify orientation for objects like ellipsoids, lines, capsules or arrows.
Example2: draw a line segment:

void drawLineSeg(mjModel *mm, mjData *dd, mjtNum p1[], mjtNum p2[], float rgba[], mjvScene *scene, const mjvOption *opt)
{
    mjvGeom *mylinegeom;
    mjtNum linesize[3] = {0., 0., 0.};
    mjtNum linepos[3] = {0., 0., 0.};
    mjtNum com_proj[3];
    mjtNum diff[3];
    mjtNum myquat[4] = {1, 0, 0, 0};
    mjtNum myrot3x3_2[9] = {1., 0., 0., 0., 1., 0., 0., 0., 1.};

    // Line start: at ith point
    mju_copy3(linepos, p1);
    // Line end: at i+1st point
    mju_sub3(diff, p2, p1);
    // line length
    linesize[2] = mju_norm3(diff);

    // set mat to minimal rotation aligning b-a with z axis
    mju_quatZ2Vec(myquat, diff);
    mju_quat2Mat(myrot3x3_2, myquat);

    // one more geom to render
    scene->ngeom = scene->ngeom + 1;
    // mygeom now points to the last location in geoms buffer
    mylinegeom = scene->geoms + scene->ngeom - 1;
    // decor geom
    mylinegeom->objtype = mjOBJ_UNKNOWN;
    mylinegeom->objid = -1;
    mylinegeom->category = mjCAT_DECOR;
    mylinegeom->segid = scene->ngeom;
    // Add it to the scene
    mjv_initGeom(mylinegeom, mjGEOM_LINE, linesize, linepos, myrot3x3_2, rgba);

    mjv_addGeoms(mm, dd, opt, NULL, mjCAT_DECOR, scene);
}

You need to call these functions after updating and before rendering the scene. For e.g., in basic.cc, you can insert the code between:

    // update scene and render
    mjv_updateScene(m, d, &opt, NULL, &cam, mjCAT_ALL, &scn);

    // ADD DECORATIVE GEOMs TO THE SCENE
    mjtNum mypt[3] = {0,0,0};
    float clr[4] = {1,0,1,0.5};
    drawSph(m, d, pt, 0.05, clr, &scn, &opt);

    mjr_render(viewport, &scn, &con);

If you use Python bindings, please check out Modifying the scene in the introductory tutorial.

[kevinzakka]

In the tutorial colab, you can find a section on adding custom geoms to a scene.

[yuvaltassa]

The two answers above refer to the 3D scene. It is also possible to add plots via the low-level API.
Unfortunately there are currently no Python bindings for the GUI elements, you'll need to use C++ for that.