Requesting clarification on camera parameters.

[sh4rvani]

I am seeking further clarification on a few aspects related to camera parameters, bird’s eye view, and Z-stabilised cameras. I have some understanding of these concepts but would appreciate it if you could provide more detailed explanations. Below are my queries and observations:

1. Understanding Camera Parameters: Could you please explain the following camera parameters in detail: pos: How do these coordinates affect the camera’s perspective? euler: What role do they play in determining the camera’s orientation? fovy: How does it impact the camera’s view? Are there methods other than fixed cameras that can be used for these applications?

2. Bird-Eye View Camera: I understand that the bird’s eye view is used in specific situations like olfaction (from the github notebooks). However, I am planning to use it for observing turning and despite adding a Z-stabilised camera (which is added in the turning notebook), and the camera name is set to as "birdeye_cam", like in this one:
   
   The video is not stabilized even if a Z-stabilized camera is added, like the one in the above with other camera parameters, and it appears blank if a Z-stabilized camera is not added.

3. Could the blank video be due to Incorrect fovy settings? or Incorrect pos adjustments? (I have tried adjusting pos and fovy multiple times but still encounter blank videos not only in this case but other cases such as olfaction too.) Are there any specific settings or adjustments I should consider to avoid blank videos?

4. Z-Stabilized Camera: I understand that Z-stabilized cameras are used in specific situations like CPG (Central Pattern Generator) modeling and turning (https://github.com/NeLy-EPFL/flygym/blob/main/notebooks/turning.ipynb). Why is the Z-stabilized camera preferred for turning maneuvers and not for other scenarios? Why do I end up getting a somewhat stabilized video when I am trying to follow the configurations without explicit camera parameters as per the turning Github notebook? like in this one:
   
   However, the video destabilised when I used the second configuration with explicit camera parameters during turning:
   
   Could you explain why the video is somewhat stabilized in the first case but destabilized in the second? Is it due to the explicit camera parameters like pos, euler, and fovy, and how do they impact?
   I also want to know why explicit parameters for the camera are not used in turning Github notebook like they are used in other notebooks, such as olfaction.

5. Could you provide a detailed explanation of the following:
   Bird’s Eye View Function: How does it work, and what are its key applications?
   Z-stabilized Camera Function: How does it stabilize the view, and why is it particularly useful in turning scenarios?

I apologise for the very lengthy question and any inconvenience caused. I would greatly appreciate it if you could address these points in detail. Your insights will help me better understand these concepts and resolve the issues I am facing with the camera setup. Thank you for your time and support !

[stimpfli]

Hi I will try to address every question as thoroughly as possible do not hesitate to come back to me if any of those answers were not clear.

• The pos parameter refers to the position of the camera with respect to the attachment_point.
• The euler parameter defines the orientation of the camera (in its initial orientation the camera points downward).
• The fovy parameter defines the field of view in angles (a higher values means the camera will span a larger field of view and will thus look more zoomed out).
• I am not sure I understand the last part of the question but those parameters can be used with any camera.

All those parameters have a MuJoCo parameters counterpart.

2. ZStabilizedCameras avoid translation along the z axis meaning that it does not move with respect to the floor. MuJoCo cameras can have different "modes", in the fixed mode the orientation and position is fixed with respect to the attachement point (the camera translates and rotates with the fly). We found those small oscillation along the z axis quite annoying when visualising our data hence the usage of Z stabilised camera. While using the birdeye camera the gain in stability will be minimal as it will just correct for the change in height of the camera with all other degrees of freedom still behaving as fixed (moving). You can either use a track camera (no changes in the orientation) or write your own camera class similar to the ZStabilizedCamera to stabilise the degrees of freedom you would like to cancel.

3. The blank videos are actually due to the settings of extent, znear and zfar. In our xml zfar is set to 3 this means that past a certain camera to object distance the object will not be rendered anymore. I would try increasing the zfar parameter in the xml at flygym/data/mjcf/neuromechfly_seqik_kinorder_ypr.xml.

4. This has to do with the mode parameter of the camera. In the config.yaml at flygym/config.yaml are defined the parameters for default cameras. Most camera modes are track meaning that the translation will be fixed with respect to the attachement point (moving with it as if it was attached to it) while the rotations will not update ("fixed" in global coordinates) hence keeping the horizon flat for instance and giving this more stabilised effect. .

5. The logic is the following:

• Every camera needs parameters
• We defined predefined cameras whose parameters are summarised in the config.yaml file. When instantiating a camera with one of those names, the parameters of this camera are used.
• The user can define any camera with whatever orientation, position ,mode and other MuJoCo camera parameters. Birdeye_cam is an example of those custom cameras as it is not part of the set of predefined cameras.
• The user can define new camera classes based on their need. We decided to create a Z-Stabilized camera class where the height of the camera is fixed with respect to the floor. Other update rules can also be defined.