ER_SEGMENTATION_MASK_OBJECT_AND_LINKINDEX interfering with mouse picking

[mbusy]

Hi Erwin,

I noticed a weird behavior while trying to wrap the generation of segmented images in the qiBullet project.

When I call the getCameraImage method with the ER_SEGMENTATION_MASK_OBJECT_AND_LINKINDEX flag, mouse picking operations behave weirdly (as if more energy was added to the constraint, see gif below).

If I use the ER_NO_SEGMENTATION_MASK flag, the mouse picking behaves fine (see gif below).

It seems similar to what your describe in #1958, more precisely in your last answer.

You can find the snippet I used to generate the gifs below. In this snippet, qiBullet is only used to spawn the Pepper robot, I exposed the camera related code in the getImageTest method. I also tested with the humanoid provided in pybullet_data/humanoid/humanoid.urdf, I obtain the same behavior

import math
import numpy as np
from qibullet import PepperVirtual
from qibullet import SimulationManager
import pybullet


LINK_PARENT_INDEX = 13
WIDTH = 320
HEIGHT = 240
NEAR_PLANE = 0.01
FAR_PLANE = 100
HFOV = 58.0
VFOV = 45.0


def getImageTest(robot, segmentation):
    _, _, _, _, pos_world, q_world = pybullet.getLinkState(
        pepper.getRobotModel(),
        LINK_PARENT_INDEX,
        computeForwardKinematics=False)

    rotation = pybullet.getMatrixFromQuaternion(q_world)
    forward_vector = [rotation[0], rotation[3], rotation[6]]
    up_vector = [rotation[2], rotation[5], rotation[8]]

    camera_target = [
        pos_world[0] + forward_vector[0] * 10,
        pos_world[1] + forward_vector[1] * 10,
        pos_world[2] + forward_vector[2] * 10]

    view_matrix = pybullet.computeViewMatrix(
        pos_world,
        camera_target,
        up_vector)

    if segmentation:
        flags = pybullet.ER_SEGMENTATION_MASK_OBJECT_AND_LINKINDEX
    else:
        flags = pybullet.ER_NO_SEGMENTATION_MASK

    projection_matrix = pybullet.computeProjectionMatrix(
        left=-math.tan(math.pi*HFOV/360.0)*NEAR_PLANE,
        right=math.tan(math.pi*HFOV/360.0)*NEAR_PLANE,
        bottom=-math.tan(math.pi*VFOV/360.0)*NEAR_PLANE,
        top=math.tan(math.pi*VFOV/360.0)*NEAR_PLANE,
        nearVal=NEAR_PLANE,
        farVal=FAR_PLANE)

    camera_image = pybullet.getCameraImage(
        WIDTH,
        HEIGHT,
        view_matrix,
        projection_matrix,
        renderer=pybullet.ER_BULLET_HARDWARE_OPENGL,
        flags=flags)

    return camera_image


if __name__ == "__main__":
    simulation_manager = SimulationManager()
    client = simulation_manager.launchSimulation(gui=True)
    pepper = simulation_manager.spawnPepper(
        client,
        spawn_ground_plane=True)

    try:
        while True:
            getImageTest(pepper, True)

    except KeyboardInterrupt:
        simulation_manager.stopSimulation(client)

[erwincoumans]

There is an option to disable mouse picking entirely, would that help?

import pybullet as p
p.configureDebugVisualizer(p.COV_ENABLE_MOUSE_PICKING, 0)

It is different from resetDebugVisualizerCamera (in the other issue), but perhaps there is some bug/inference between mouse pick and getCameraImage.

[mbusy]

I indeed saw that mouse picking could be disabled, but it's a cool feature and I'd rather keep it. I wondered if the two could be connected, I'll eventually investigate that, see if I can avoid that interference.

Even though its a different method, do you know why the resetDebugVisualizerCamera call interferes with mouse picking ? It could be a good clue to understand the problem at hand.