Split the SRanipalModule in two components to be run in the two networks separately

[S-Dafarra]

Right now, the SRanipalModule needs to run on the operator side since it needs to access the data coming from the headset. At the same time, it needs to open the head remote control board to control the gaze. In case of delayed network, connecting to the control board could be expensive. Hence, it would be better to separate the parts that connect to the headset from the part that connects to the robot.

[S-Dafarra]

Considering that the SRanipal SDK has been deprecated, it would also be better to use these extensions:

• https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#XR_EXT_eye_gaze_interaction
• https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#XR_HTC_facial_tracking

[S-Dafarra]

Considering that the SRanipal SDK has been deprecated, it would also be better to use these extensions:

• registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#XR_EXT_eye_gaze_interaction
• registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#XR_HTC_facial_tracking

Unfortunately, neither these two seem supported by SteamVR yet. See https://github.khronos.org/OpenXR-Inventory/extension_support.html#valve_steamvr

[S-Dafarra]

Unfortunately, neither these two seem supported by SteamVR yet. See github.khronos.org/OpenXR-Inventory/extension_support.html#valve_steamvr

Nonetheless, if I list the extensions from the OpenXr device, I do see them. Maybe they are added by the VIVE Console application

[S-Dafarra]

Nonetheless, if I list the extensions from the OpenXr device, I do see them. Maybe they are added by the VIVE Console application

This is the output I see

[INFO] |yarp.device.openxrheadset| Supported extensions:

    - XR_EXT_hand_tracking
    - XR_HTC_hand_interaction
    - XR_EXT_eye_gaze_interaction
    - XR_HTC_facial_tracking
    - XR_HTC_vive_srworks_pass_through
    - XR_MSFT_scene_understanding
    - XR_HTC_passthrough
    - XR_KHR_vulkan_enable
    - XR_KHR_vulkan_enable2
    - XR_KHR_D3D11_enable
    - XR_KHR_D3D12_enable
    - XR_KHR_opengl_enable
    - XR_KHR_win32_convert_performance_counter_time
    - XR_EXT_win32_appcontainer_compatible
    - XR_KHR_binding_modification
    - XR_KHR_composition_layer_depth
    - XR_KHR_visibility_mask
    - XR_EXT_active_action_set_priority
    - XR_EXT_dpad_binding
    - XR_EXT_frame_composition_report
    - XR_EXT_hand_tracking_data_source
    - XR_EXT_hand_joints_motion_range
    - XR_EXT_hp_mixed_reality_controller
    - XR_EXT_local_floor
    - XR_EXT_palm_pose
    - XR_EXT_uuid
    - XR_FB_display_refresh_rate
    - XR_HTC_vive_cosmos_controller_interaction
    - XR_HTC_vive_focus3_controller_interaction
    - XR_HTC_vive_wrist_tracker_interaction
    - XR_MND_headless
    - XR_VALVE_analog_threshold
    - XR_HTCX_vive_tracker_interaction
    - XR_EXT_debug_utils

Moreover, if I click on MANAGE OPENXR API LAYERS in the SteamVR settings I see

that seems suggesting they are available.

[S-Dafarra]

I started adding the facial expression extension in gbionics/yarp-device-openxrheadset@e73e989

I still need to expose the weightings and test it.

[S-Dafarra]

I still need to expose the weightings

Done in gbionics/yarp-device-openxrheadset@2aa1f20

I also added the gaze extension with gbionics/yarp-device-openxrheadset@d283165 and gbionics/yarp-device-openxrheadset@ebf35b5

Related PR: gbionics/yarp-device-openxrheadset#45

I still need to test the whole thing.

[S-Dafarra]

Today I started testing it. Overall, the results are not satisfactory.

1. I am not able to get the eye expressions. This code failed with VALIDATION_FAILURE and I am not able to find the root cause.
2. For the gaze, the query to detect whether the gaze is supported always return false. If I ignore this check, I am able to get the gaze. But:
   • the VR performances seems completely degraded
   • the gaze position appears to be only a few mm away with respect to the head. This makes me think that I am not able to get the fixation point, but only the gaze direction. This is a bit of a bummer because it makes it impossible to control the vergence in the robot eyes.

[S-Dafarra]

• I am not able to get the eye expressions. This code failed with VALIDATION_FAILURE and I am not able to find the root cause.

It was my fault. The fix is in gbionics/yarp-device-openxrheadset@9b3d2b3. Basically I was using in the wrong way the OpenXR handles.

• the VR performances seems completely degraded

I tried again today and it did not seem so problematic. Probably a reboot was enough

[S-Dafarra]

• This makes me think that I am not able to get the fixation point, but only the gaze direction.

I moved to publishing the gaze direction in gbionics/yarp-device-openxrheadset@5cd1ac1

[S-Dafarra]

Apparently, it is no more possible to download the SRanipal SDK, so this is now necessary

[S-Dafarra]

In the meantime, we merged gbionics/yarp-device-openxrheadset#45

[S-Dafarra]

While the interface on the VR side is ready, it would be necessary to develop the part on the application side (i.e. edit the SRanipalModule) to use the interface.

One thing that will be lost is the capability of controlling the vergence, since the extension only gives a single fixation point, while before I could access the fixation point of both eyes.