Perspective transformation from 3D point to pixel position

[lopsided]

I have a scene set up with a camera pointing down at the origin, like

'to_world': T.look_at(
                origin=[0, 0, 10],
                target=[0, 0, 0],
                up=[0, 1, 0]
            )

Now I just want to do a quick perspective transformation from some 3D points in my scene to see where they may appear in the image. I've been trying to do this by extracting the projection matrix with:

params = mi.traverse(scene)
camera = scene.sensors()[0]
film = camera.film()
prj = mi.perspective_projection(film.size(), film.crop_size(), film.crop_offset(), params['sensor.x_fov'], params['sensor.near_clip'], params['sensor.far_clip'])

and then applying to a point like:

uv = prj @ mi.Point3f(np.ones(3))

but this is as far as I've got. I don't know how to convert the 3D point that's returned back to an image position, and also using np.zeros(3) as the point gives me [nan, nan, -inf] back, which I wasn't expecting as this should be in the centre of my view.

Any suggestions would be very welcome!!

[njroussel]

Hi @lopsided

Have a look at this discussion: #1138
I believe you're looking for something similar (slightly simpler).

[lopsided]

Thanks @njroussel ! I had missed the sensor.world_transform().inverse() bit from that message first time around. It feels a little clunky, but this seems to work ok...

def _project_to_image(self, points: torch.Tensor) -> torch.Tensor:
    """
    Project 3D points to the image plane.
    """
    params = mi.traverse(self.scene)
    sensor = self.scene.sensors()[0]
    film = sensor.film()

    # Create the projection matrix
    prj = mi.perspective_projection(film.size(), film.crop_size(), film.crop_offset(), params['sensor.x_fov'],
                                    sensor.near_clip(), sensor.far_clip())

    # Get the inverse camera world transform
    wti = sensor.world_transform().inverse()

    # Project the points
    if points.ndim == 1:
        points = points[None, :]
    points_homogeneous = torch.cat([points, torch.ones(len(points), 1, device=points.device)], dim=1)
    hpp = points_homogeneous @ torch.tensor((prj @ wti).matrix, device=device)[0].T
    pp = hpp[:, :3] / hpp[:, 3][:, None]
    dim = torch.tensor(film.crop_size(), device=device)[None, :]
    uv = pp[:, :2] * dim
    return uv