Custom GLContext python ? #1990

GCrequer · 2024-05-13T17:35:59Z

GCrequer
May 13, 2024

Hello everyone,

I'm a student in robotics and computer vision, and I'm trying to get into MuJoCo. I have some knowledge of OpenGL, enough to understand the basics, but not enough to be fully independent with all the subtleties.

My goal is to simulate cameras with strong distortions (fisheye, 360°, etc.) embedded on robots. To do this, I need to bypass MuJoCo's default OpenGL context. I've read that it's entirely possible to do so here and I've found the class template for it in the __init__.py files (for now, I'm just trying with mujoco.GLContext as it's integrated).

However, it's mentioned that once the context is created, the classic mjr_ functions should work. So, I don't know where my mistake is in the following simplified code:

import mujoco as mj
import numpy as np

from mujoco.glfw import glfw
from matplotlib import pyplot as plt

xml= """
<mujoco>
    <asset>
    <texture type="2d" name="groundplane" builtin="checker" mark="edge" rgb1="0.2 0.3 0.4" rgb2="0.1 0.2 0.3" markrgb="0.8 0.8 0.8" width="300" height="300"/>
    <material name="groundplane" texture="groundplane" texuniform="true" texrepeat="5 5" reflectance="0.2"/>
    </asset>
	<worldbody>
        <light pos="0 0 1.5" dir="0 0 -1" directional="true"/>
            <geom name="floor" size="0 0 0.05" type="plane" material="groundplane"/>
		<body pos="0 0 1">
			<joint type="free"/>
			<geom type="box" size=".1 .2 .3" rgba="0 .9 0 1"/>
		</body>
	</worldbody>
</mujoco>
"""

model = mj.MjModel.from_xml_string(xml)
data = mj.MjData(model)

cam = mj.MjvCamera()

opt = mj.MjvOption()
mj.mjv_defaultOption(opt)
scene = mj.MjvScene(model, maxgeom = 100)


### TO ADAPT : changing shaders, projection matrix, etc.
ctx = mj.GLContext(1200, 900)
ctx.make_current()
###


frame1 = np.zeros((900, 1200, 3), dtype=np.uint8)
while True:
    mj.mj_forward(model, data)
    mj.mj_step(model, data)
    
    viewport = mj.MjrRect(0, 0, 1200, 900)

    mj.mjv_updateScene(model, data, opt, None, cam,  mj.mjtCatBit.mjCAT_ALL.value, scene)
    mj.mjr_render(viewport, scene, ctx)
    mj.mjr_readPixels(frame1, None, viewport, ctx)
    plt.imshow(frame1)

    plt.pause(0.01)

plt.show()

I prefer to use my own context because making changes to mjrContext is not recommended here,and I believe that "everything" is possible with our own context, and that it's through this GLContext class that we can create it. I know that theses classes are only intended for offscreen rendering (#544), I will have to postprocess my frames so it is not a problem.

If you have any hints to give me, or an alternative to suggest (knowing that I'm already struggling to understand how to modify shaders and retrieve vertices in this system), it would be great to know !
Thanks in advance!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Custom GLContext python ? #1990

Uh oh!

{{title}}

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Custom GLContext python ? #1990

Uh oh!

GCrequer May 13, 2024

Replies: 0 comments

GCrequer
May 13, 2024