About cameras in batched environments

[eratc]

Is there a better way to record batched environments other than to add a bunch of cameras with add_camera, and have large env_spacing so they don't record robots from other environments? You also need to be mindful of the coordinates when you change the batch size, since coordinate system for cameras are not relative to the environments. If there is a method to do this which is fast with parallelization and less fiddly, or there is a plan to add one, I'd be glad to know!

[zhouxian]

Yes! We are actually done with prarallel rendering and working on some test and optimization

[Kashu7100]

@zhouxian I think it is #545

[JoseBarreiros-TRI]

This is great, thank you.
I just tested the feature by running the single_franka_envs.py example and it works great. I noticed that the camera pose can only be set in global coordinates when using batched environments; however, this drastically limits the use cases. In the image below, the cameras render each environment correctly but are positioned in different poses even though we can only control a single pose. Having a way to define the camera pose in env local coordinates will be ideal.

Also, note that the forces from env0 are rendered in env1.

[ACMLCZH]

Thank you for your suggestion and bug report! Specifying environment-specific camera poses is on our schedule. And force mismatches occur because the renderer currently maintains the force mesh only for env0. This issue will be addressed in future pull requests.

[eratc]

For more specific use cases, setting cam poses separately for each environment might be needed, but in this case, you can build the scene with zero env_spacing for unified view of all environments.

[JoseBarreiros-TRI]

this is a good idea, thanks

[eratc]

Thanks, looking forward to it! Do you also plan to release motion planning support in batched environments soon?

[vlamirand]

Hi! How is the progress on the ability to set camera poses individually for each environment in batched mode? Have you been able to implement this feature yet, as well as the ability to attach the camera to a body (e.g., the robot's end-effector, as discussed at the beginning of the thread)? These are very important features for our tasks. Thank you!

[vlamirand]

(Just adding to my previous comment)
I've looked into the code and managed to change the camera position based on a body's pose. Specifically, I found the relevant parts in Genesis/genesis/ext/pyrender/renderer.py, within the render function and then in _forward_pass. Inside _forward_pass, the line cam_pos = scene.get_pose(scene.main_camera_node)[:3, 3] gets the camera position. You could, in theory, override cam_pos here to dynamically adjust the camera based on an object's pose.

However, I discovered that the performance of this approach is very low, and not sufficient for our needs. To achieve the required speed, it seems like the entire rendering process would need a complete overhaul. I suspect this is the reason why implementing per-environment camera control is taking so long – it likely requires significant changes to the underlying rendering pipeline. It seems like the options are either to drastically simplify the rendering or to avoid using this renderer for online training scenarios.