ER_TINY_RENDERER freezes when camera exactly on joint

[bjuergens]

Problem: When the camera is on the joint of a robot, the rendering is very slow and seems to enter an infinity loop eventually

Workaround/Solution: Move the camera back a little until it's outside the robot

The issue on it's own is not a big deal, since a workaround exists. But I can imagine, that it can help to trouble-shoot other issues involving random freezes and low speed. If it doesn't, feel free to close the issue and forget about it.

Observations:

• the freeze seems to happen after the robot touches the ground the first time.
• it doesn't happen with opengl-renderer, only with tiny-renderer
• it happens in DIRECT and as well as in GUI mode
• I have waited for 5 minutes before assuming an infinity loop
• I have tried out different view_matrices and camera_directions. They seem to change the time until the freeze, but it always seems to happen eventually
• changing the the camera-positions to outside the robot seems to be the only reliable workaround

example:

here is an example project, that initiates a simple world with a husky-robot, where an image is generated by a camera, that is on a joint of the robot. The Script prints the number of the current iteration on each iteration, and it prints a short message before and after it calls getCameraImage. It seems to enter an infinite loop on frame 30 (if I run it directly on my notebook) or frame 150 (if I run it inside docker). To stop the Script, you'll need to enter sudo pkill python3 in another window. Due to the infinity-loop the container can't stop on it's own.

 cat freeze.py 
#!/usr/bin/env python3
import numpy as np
import pybullet as p
import pybullet_data

class Husky(object):
    def __init__(self):
        self.husky_model = p.loadURDF("husky/husky.urdf",
                                      basePosition=[0, 0, 0.1],
                                      baseOrientation=p.getQuaternionFromEuler([0, 0, 0]))

        for joint in range(p.getNumJoints(self.husky_model)):
            if p.getJointInfo(self.husky_model, joint)[1] == b'user_rail':
                self.zed_camera_joint = joint

    def update_cam(self):
        joint_pos, joint_orn, _, _, _, _ = p.getLinkState(self.husky_model, self.zed_camera_joint,
                                                          computeForwardKinematics=True)
        cam_rot = np.array(p.getMatrixFromQuaternion(joint_orn)).reshape(3, 3)
        cam_direction = cam_rot.dot((1, 0, 0))
        cam_up_vec = cam_rot.dot((0, 0, 1))
        cam_pos = joint_pos
        # no freeze, when camera pos is slighty behind joint
        # cam_pos = joint_pos - (cam_direction * 1.0)
        cam_target = joint_pos + (cam_direction * 1.0)
        view_mat = p.computeViewMatrix(cam_pos, cam_target, cam_up_vec)
        proj_mat = p.computeProjectionMatrixFOV(100, 80 / 80, 0.1, 100)
        print("getCameraImage...")
        img = p.getCameraImage(80, 80, viewMatrix=view_mat, projectionMatrix=proj_mat, renderer=p.ER_TINY_RENDERER)
        print("getCameraImage... done")
        return img


physicsClient = p.connect(p.DIRECT)

p.setAdditionalSearchPath(pybullet_data.getDataPath())
p.setGravity(0, 0, -15)
p.loadURDF("plane.urdf")

my_robot = Husky()

step = 0
while 1:
    step = step + 1
    print(step)
    if step % 10 == 0:
        my_robot.update_cam()
    p.stepSimulation()

reproduction:

To rule out the possibility, that there is something wrong with my setup, I prepared a Dockerfile to reproduce the isse:

cat Dockerfile 
FROM ubuntu:bionic
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y python3-pip python3-dev python3 git && pip3 install numpy
RUN apt-get install -y cmake

RUN git clone --recurse-submodules https://github.com/bulletphysics/bullet3.git 
RUN cd bullet3 \
 && ./build_cmake_pybullet_double.sh \
 && python3 -m pip install .

ADD freeze.py freeze.py
CMD python3 freeze.py

you can build it and run the example with:

docker build -t test . && docker run --rm -it test

(assuming freeze.py and Dockerfile are in your current directory):

[bglima]

Very informative issue! Thanks. I experienced the same problem.