Improved Features and Outlook

[fremaxter]

Hi,
I'm impressed with what you have already accomplished, good job and awesome to have a potentially new open source contender for a physics simulator, challenging Isaac Sim / Isaac Lab. I hope this project can gain some traction and further development to create something powerful and become the go-to for researchers or commercial.
I like the code structure and easy way to get started, while still delivering performance and many features.

I would like to give some feedback, and invite others, on what features would improve the use cases for applications. If I have missed a feature or did not use the framework as currently designed, I would like to apologise in advance and I'm looking forward to some insights and explanations. Could you maybe give some rough estimates or updates on when such features could be developed? Such communication is very helpful to keep the community engaged.

Visual Learning:

• For doing any visual learning tasks (e.g. RL) high-speed image generation is crucial. Currently, it seems that the camera cannot be parallelized across the envs (it renders all envs and doesn't separate them). The roadmap names "Tiled rendering" which hopefully includes env separation and batched high-performance rendering. Really looking forward to this so Genesis can be used for such applications.
• Currently, I only get around 1fps (on my 3090) running the render demo file. It's great to have the option for photorealistic rendering, but for training visual learning tasks it would be great to have an additional option that has greater visual quality than the rasterization but faster performance than the current Luisarender ray tracer. (You may have hinted at a feature like this for the generative framework, but the physics simulation use of Genesis needs this just as necessary.) It would be great if the visual properties could be somewhat united during this feature upgrade. The API seems to differentiate currently that many features only work for ray tracer (e.g. most materials) and other for rasterization. It would be great if the combined renderer can include e.g. features like different lights and materials as well, to actually make them viable for large-scale visual learning / manipulation tasks. (high priority)
• Do you know how Isaac Lab currently does this? They seem to have some sort of ray tracing even for parallel envs with decent performance if I'm not mistaken.
• What do you think of the Madrona renderer now used in DeepMind Playground Mujoco MJX? They seem to claim great performance for ray-traced batched image generation.

Sensors:

• The camera is the most important sensor for robotics and AI, but other sensors, most importantly: lidar / ray caster would be a valuable addition to cover the most important use cases. (medium priority)

Physics Solvers:

• It seems to me that currently only rigid bodies can be parallelized into multiple envs. For training policies of e.g. robotic with deformables or fluids: would it also improve performance to create multiple envs for such setups? (lower priority)

Could you please give a general update about the future development of Genesis. You mentioned: "we will be back with full thrust to resume developing Genesis with an engineering team in Feb". How did this project come to life and how big do you think it can become? Could you maybe elaborate on what and who the mentioned engineering team is. Who supports it (which companies or money) or is it a team of voluntary engineers?

I'm in contact with many researchers and we would be interested in the future development effort of this physics simulation framework and if it's worth it for us to invest time and effort into it. We are currently deciding between MJX, Maniskill, Isaac Lab, and Genesis. We appreciate the open-source nature and impressive results and intuitive code base of Genesis. However, each simulation framework has its own API and we would like to focus on one and try to improve it instead of switching efforts between them because of lack of further development. What is your honest future direction of Genesis, is it a proof of concept for a simulation and generative framework, or are you planning on creating a sustainable and ongoing development of the physics simulation?

We are looking forward to your insights and comments and are inspired to create something great!

[fremaxter]

Could you please give a general update about the future development of Genesis. You mentioned: "we will be back with full thrust to resume developing Genesis with an engineering team in Feb". How did this project come to life and how big are its plans? Could you maybe elaborate on what and who the mentioned engineering team is. Who supports it (which companies or money) or is it a team of voluntary engineers?

Also, some more update about the status of current developments and updates on the roadmap would be good to keep community engaged.

@Genesis-Embodied-AI announced such updates for February, could you please keep us up to date.