Declaring (SDF) plugin(s) using MuJoCo's Python API

[joaocandre]

Intro

Hi!

I am a researcher using MuJoCo to simulate contact dynamics in object manipulation (hollow bodies) with humanoid robots.

My setup

Mujoco 3.3.2, Python, Linux

My question

I'm building a simulation model programmatically through the mjSpec Python interface, and I'm having trouble properly loading the sdflib plugin.
A minimal model defined as

import mujoco

spec = mujoco.MjSpec()
spec.add_mesh(name='some_mesh', file='mesh.stl')
body = spec.worldbody.add_body(name='some_body', pos=[0.0, 0.0, 1.0], mass=10)
geom = body.add_geom(name='some_geom', type=mujoco.mjtGeom.mjGEOM_SDF, meshname='some_mesh', rgba=[0.1, 0.1, 0.1, 1.0]) 
body.add_freejoint()

with open('model.xml', 'w') as file:
    file.write(spec.to_xml())
    file.close()

will lead to a runtime Segmentation Fault on any collision with other free bodies in the simulation.
Looking into the sdf examples, the sdflib plugin would need to be explicitly declared on the model file e.g.

<extension>
    <plugin plugin="mujoco.sdf.sdflib">
        <instance name="sdf">
            <config key="aabb" value="0"/>
        </instance>
    </plugin>
</extension>

and the corresponding geometry updated to

<geom name="some_geom"  type="sdf"  mesh="some_mesh" rgba="0.1 0.1 0.1 1">
    <plugin instance="sdf"/>
</geom>

and, indeed, when manually updating the resulting XML leads to a functional simulation (i.e. working non-convex collisions).

However, I'm unable to replicate this using the Python API: adding the plugin with

spec.add_plugin(name='sdf', plugin_name='mujoco.sdf.sdflib')

will cause an unrecognized plugin 'mujoco.sdf.sdflib' error when exporting the model to XML.
Conversely, attempting to load the plugin library beforehand:

mujoco.mj_loadPluginLibrary('/lib/mujoco.mujoco_plugin/libsdf.so')

will output plugin 'mujoco.sdf.bolt' is already registered and thus I assume the library is already loaded and the sdflib plugin should be available.
Is this a limitation of the Python bindings, or is there an alternative way to specify the plugins?

Minimal model and/or code that explain my question

No response

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[quagla]

Hi, the plugin sdflib has been removed and the functionality is now supported natively in mujoco. You only need to declare the geom type as sdf. It sounds like you are using the development version of mujoco in python (which has this change) and the 3.3.2 for your XML (which hasn't).

[quagla]

Are you able to run all of the examples in the sdf folder using the dev version?

[joaocandre]

No, it fails with plugin <...> not found.

(MuJoCo 3.3.4, built locally)

[quagla]

I can load it fine (albeit internally, I didn't try to clone from github), so I'm afraid there is some issue with your installation..

[joaocandre]

It does seem that way, as the prebuilt 3.3.4 binaries are able to load the example models. I'll check my build flags to see if anything relevant may be missing.
Nevertheless, even this version of simulate (3.3.4 prebuilt) crashes in my particular case, this time with mjc_getSDF: Plugin is not a signed distance field at slot 0. The plugin libraries seem to be correctly loaded:

mujoco_error2.mp4

If I, as per the original post, append the explicit declaration of the plugin in the model file, I can get it to simulate properly:

mujoco_working_334.mp4

[joaocandre]

Building from the latest sources (3.3.5), and any issues seem to have been resolved:

mujoco_working_335.mp4

which I assume, given by the absence of Plugins registered by library libsdf_plugin.so messages, that fully implements the SDF natively (also noticeably faster loading).

[quagla]

Happy that your issue was solved. I will close this discussion and feel free to post in #2643 about performance.

[joaocandre]

For reference, this issue was solved by updating to the latest commit on the main branch, but the root issue is still present, at least until 3.3.5 is released:
it's not clear how to declare the SDF plugin(s) using MuJoCo's Python API, and I haven't found anything pertaining to this in the documentation or tutorial notebooks.
Building 3.3.3 or 3.3.4 from source (core libraries, binaries and Python bindings) I still get the aforementioned errors.
In practice, latest development commit fixes this particular issue but is understandably unstable (I'm getting crashes when attaching models/bodies, for instance) .

[quagla]

Well 3.3.3 and 3.3.4 will never change, they are frozen in time. What crashes are you getting with dev? Can you please open an issue with a minimal example. We will probably release 3.3.5 this week so I would like it not to crash.

[joaocandre]

Simulation crashes when attaching a robot model (Apptronik Apollo) to a top-level frame; and, quite strangely, when adding a light to the worldbody.

I'll gather more details and maybe open an issue.

[quagla]

For the python API, is this not helping? https://github.com/google-deepmind/mujoco/blob/main/python/mujoco/specs_test.py#L876

[joaocandre]

That does seem to be proper syntax for using the plugin libraries (see below why it was failing in my case) and in my opinion a reference should made to this on the Python documentation page

I have, however identified the underlying cause of these issues, which is basically the lack of the plugin libraries where the tool binaries and Python bindings expect them to be: in the first, <bin>/mujoco_plugin/libsdf_plugin.so and in the second <venv>/lib/pythonXXX/site-packages/mujoco/plugin/libsdf_plugin.so.

When building MuJoCo from source, there are no CMake install rules for the plugin libraries, in contrast with how fixed releases are packaged; similarly, the package with the Python bindings only includes the core library.
I've submitted a PR addressing the CMake configuration part, but the pip package built from sources (follow the instructions) still skips this.

[quagla]

Acknowledged, we need better docs. Thanks for the PR!