Unprojecting depth maps to 3D-world coordinates

[a-3wais]

Hi!
I have a bunch of rendered RGBA images and depth maps of the same scene that were rendered by the same camera from different locations (same Intrinsics different Extrinsics) using the AOV integrator.
I used something like this snippet for rendering the dataset.

scene = mi.load_dict({
    'type': 'scene',
    # The keys below correspond to object IDs and can be chosen arbitrarily
    'integrator': {'type': 'aov',
                    'aovs': 'dd.y:depth,nn:sh_normal',
                    'my_image': {'type': 'direct',}
                },
    #'integrator': {'type': 'direct'},
    'light': {'type': 'constant'},
    'garment': {
        'type': 'obj',
        'filename': obj_file,
        'to_world': T.scale(0.03),
        'bsdf': {'type': 'twosided',
                'material': {
                    'type': 'diffuse',
                    'reflectance': {
                        'type': 'rgb',
                        'value': sampled_color
                    }
                }
            },
        },
    })

for indx, origin in enumerate(points_on_sph):
        sen = mi.load_dict({
                'type': 'perspective',
                'fov': 30,
                'to_world': mi.ScalarTransform4f.look_at(
                    origin=origin,
                    target=[0, -1.5, 0],
                    up=[0, 1, 0]
                ),
                'sampler': {
                    'type': 'multijitter',
                    'sample_count': 64
                },
                'film': {
                    'type': 'hdrfilm',
                    'width': 512,
                    'height': 512,
                    'rfilter': {
                        'type': 'tent',
                    },
                    'pixel_format': 'rgba',
                    "component_format": 'float32'
                },
            })
        im = mi.render(scene, spp=256, sensor=sen)
        im = mi.Bitmap(im, channel_names=['R', 'G', 'B', 'A'] + scene.integrator().aov_names())
        channels = dict(im.split())
        np.save(f'{out_folder}'.replace('img', 'depth') + '%05d.npy'%indx, np.array(channels['dd.y']))
        channels['my_image'].convert(component_format=mi.Struct.Type.UInt8, srgb_gamma=False, pixel_format=mi.Bitmap.PixelFormat.RGBA).write(f'{out_folder}' 
                                                                                                                                               + '%05d.png'%indx)

I have the cam2world matrix by doing this

'to_world': mi.ScalarTransform4f.look_at(
                    origin=origin,
                    target=[0, -1.5, 0],
                    up=[0, 1, 0]
                )

And the Projection matrix by doing this

x = mi.perspective_projection(sen_params['film.crop_size'], sen_params['film.crop_size'], sen_params['film.crop_offset'] , sen_params['x_fov'], sen_params['near_clip'], sen_params['far_clip'])

which gives this matrix lets call it P

P =
[[-1.86603, 0, 0.5, 0],
  [0, -1.86603, 0.5, 0],
  [0, 0, 1, -0.01],
  [0, 0, 1, 0]] 

Now I want to get the 3D world coordinates from the depth image and then reproject to another camera (for some kind of consistency loss I am developing)
I understand that the projection from 3D to 2D was done as
2D_depth = cam2world x P x 3D_point
and 2D_depth is [x,y,depth] and x,y are the pixel indices.
so, I am doing the reverse by
P_inv x cam2world_inv x 2D_depth = 3D_point

but I am not getting the correct values. they are far from the original values, not even in the same range. What exactly am I doing wrong here? Is there a better and easier way to do the inverse projection? for example in pytorch3d they provide this utility function.

   cam = cameras.FoVPerspectiveCameras(znear=0.001, zfar=10000.0, aspect_ratio=1.0, fov=30.0, degrees=True,
                                        R=np.array([cam1[:3,:3],]), T=np.array([cam1[:3,3],]))
   w1 = cam.unproject_points(torch.from_numpy(xy_depth).float())

plugging the same numbers in this pytorch3d function also don't give the correct world coordinates, different conventions I suppose.

[njroussel]

Hi @a-3wais

We do not provide any utility functions for something like this. I also don't think we have an example of a similar computation.

I can't tell you exactly what's wrong with your setup. However, debugging it should be pretty easy. The aov integrator can also output 3D positions with the position tag. This should help you validate the projection on a single view. In addition, you should render with spp=1 and the rfilter to box. Without these changes, your rays will average depth values in pixels.