Large Scale Multiprocess EGL Offline Rendering

[aseembits93]

Hi,

I am training some reinforcement agents with ~30 parallel simulations which include rendering an image of the scene. I'm using EGL for headless rendering. Here's the pseudo code for the render function which returns the rgb frame.

draw(vis_obj)
{
    eglMakeCurrent(vis_obj->eglDpy, EGL_NO_SURFACE, EGL_NO_SURFACE, vis_obj->eglCtx);
    mjv_updateScene(vis_obj->m, vis_obj->d, &vis_obj->opt, &vis_obj->pert, &vis_obj->cam, mjCAT_ALL, &vis_obj->scn);
    mjr_render(vis_obj->viewport, &vis_obj->scn, &vis_obj->con);
    mjr_readPixels(vis_obj->rgb, NULL, vis_obj->viewport, &vis_obj->con);
    return vis_obj->rgb;
}

There are 30 such vis_obj objects with each one on a different core(I'm using ray for distributing the data collection)
Somehow, I'm running into high RAM memory usage over time (around 64gigs in 48hours). It's hard to understand where all the memory is going since I'm using the compiled binary in python using python ctypes. I was wondering if I could any help regarding this. One thing I tried to alleviate the memory issue in the past is that I create entirely new simulations in every iteration of data collection and destroy them once that iteration is over, I found out that this approach leads to unreliable behavior and the code breaks at the point where a new EGL context is getting created. Looking forward to some help and guidance, Thanks!

[aseembits93]

Careful monitoring of RAM usage and re-instantiating contexts less frequently seemingly solves the problem.

[saran-t]

Hey, I'm really sorry that I missed this thread. Good to hear that you've found your way around the issue, but I'd still be keen to hear what usage pattern caused the memory blowup. Did you call mj_freeContext each time on the old context before you reinstantiate? I've been trying to track down memory leaks in MuJoCo over the past few weeks and have identified a few, but nothing that would explain 12KB/physics/second.

[aseembits93]

Hi @saran-t I do call mj_freeContext before re-instantiation but it doesn't help with the memory leakage much(documentation says that mjr_makecontext calls mjr_freecontext internally anyway?). I've noticed that re-instantiating too often leads the code to break unexpectedly. Would love to know more about what you find out!